Feature #1742

Feature #44: Implement basic introspection support

Implement introspection event sender

Added by J. Moringen over 7 years ago. Updated almost 7 years ago.

Status:ResolvedStart date:03/24/2014
Priority:NormalDue date:03/24/2014
Assignee:J. Wienke% Done:

100%

Category:Java
Target version:rsb-0.11

Description

  • Basic participant information
  • Echo server for clock measurements

Associated revisions

Revision 9ad512c2
Added by S. Wrede almost 7 years ago

  • reviewed, refactored and cleaned-up code basis from
    feature-introspection branch
  • introduced a ProtocolHandler class encapsulating the introspection
    network protocol actions
  • added an IntrospectionModel class encaspulating all the introspection
    data
  • updated the ParticipantObserver class to match new API
  • added code to allow lazy initialization of the ProtocolHandler to
    prevent recursive factory calls
  • extended DefaultConverters with Introspection converters
  • added stub classes for host and process info for testing
  • small clean ups and refactorings
  • refs #1742

Revision 8f9c7873
Added by S. Wrede almost 7 years ago

  • added abstract getType method to Participant base class
  • added implementations to all participant implementations, meaningful
    is currently mainly for Informer participant
  • added to Hello message serialization if an actual type is reported
  • refs #1742

Revision 8abc8f82
Added by J. Wienke almost 7 years ago

Implement introspection support

Implements introspection support for java.

fixes #1742
fixes #1869
refs #1978
refs #1837
refs #1823

History

#1 Updated by J. Moringen over 7 years ago

  • Description updated (diff)

#2 Updated by J. Moringen over 7 years ago

  • Assignee set to S. Wrede

#3 Updated by J. Moringen over 7 years ago

  • Assignee changed from S. Wrede to S. Sharma

#4 Updated by S. Sharma over 7 years ago

  • % Done changed from 0 to 10

#5 Updated by S. Sharma over 7 years ago

  • % Done changed from 10 to 20

- created basic structure
- divided model.cpp to hostinfo.java , partipantinfo.java , etc
- done with extracting mac address for host info - yet to test
- working on methods of informationsender.java , 20 % done yet
Question :
Do I need to attend the thursday meeting ?

#6 Updated by J. Moringen over 7 years ago

Suchit Sharma wrote:

Question :
Do I need to attend the thursday meeting ?

Only if you want. It is not clear whether there will be a meeting today, anyway.

#7 Updated by S. Sharma over 7 years ago

the function "bye" in introspectionsender.cpp calls rsb::protocol::introspection::Bye , do we need to implement this class rsb::protocol::introspection::Bye i java as well . If yes , could you please send me the cpp code for it for reference.

#8 Updated by J. Wienke over 7 years ago

This is automatically generated from the protobuf files that are part of the rsb-protocol project. If you use the correct version of the protocol, you should automatically have an implementation for that class in java as well.

#9 Updated by J. Moringen over 7 years ago

  • Status changed from New to In Progress

#10 Updated by S. Sharma over 7 years ago

  • % Done changed from 20 to 40
  • How can I test the introspection client , do I nned to make some java version of plugin.cpp(in introspection package in c++ version) or is there any other way ?
  • There are few rsc references in c++ code like:
    void handle(EventPtr query) {
    if (query->getType() == rsc::runtime::typeName<void>()) { .. }
    }

    I could not understand what is the class of query compared to here(what does rsc::runtime::typeName<void>() return) ?
  • The Logger has many states fine, finer etc . Which ones have to be used in which case ? In c++ there is a RSCDEBUG but I dont see LOG.debug in java so should these messages go to LOG.fine()

#11 Updated by J. Wienke over 7 years ago

I'll try to answer whatever I can.

Suchit Sharma wrote:

  • There are few rsc references in c++ code like:
    void handle(EventPtr query) {
    if (query->getType() == rsc::runtime::typeName<void>()) { .. }
    }

    I could not understand what is the class of query compared to here(what does rsc::runtime::typeName<void>() return) ?

Your example would check whether the data included in the event are of type void. This is done by a string comparison due to the lack of real reflection capabilities in C++. typeName returns a string representation for the type that is given in the template argument, in this case <void>.

As java has real reflection capabilities, there is no string declaring the data type. Instead, Class objects are used. So a similar comparison in java would be

event.getType().equals(Void.class)

In C++, your variable query is also just an rsb event.

  • The Logger has many states fine, finer etc . Which ones have to be used in which case ? In c++ there is a RSCDEBUG but I dont see LOG.debug in java so should these messages go to LOG.fine()

There are no real rules that we use. Using fine for debug sounds ok. Finer would probably relate to TRACE in C++.

#12 Updated by J. Moringen over 7 years ago

Suchit Sharma wrote:

  • How can I test the introspection client , do I nned to make some java version of plugin.cpp(in introspection package in c++ version) or is there any other way ?

If your code already sends introspection events, the RSB logger should be able to display these events. If your code already sends correct introspection events, we can test whether it works with the introspection display tool.

We can schedule another meeting if you are not sure how to proceed.

#13 Updated by S. Sharma over 7 years ago

I impelemented introspection.java according to RSB protocol specification ,and still facing some issues while testing , can we schedule a meeting to verify this and decide on how to proceed.

#14 Updated by S. Sharma over 7 years ago

I was again getting spread connection error after a few minor changes(which should not be the cause) , the port is commented in rsb.conf so that socket connection is used , below is the error in QueryHandlerTest.java , it occurs when testListner(Listner to the queue of events is activated) :
May 23, 2014 1:04:15 PM rsb.transport.spread.SpreadWrapper makeConnection
INFO: reoccuring +SpreadException+ during connect to daemon: Socket(): java.net.ConnectException: Connection refused

may be it is still trying to connect to spread.

#15 Updated by S. Wrede over 7 years ago

Are you 100% sure that no further rsb.conf is involved? Please also check your home directory: ~/.config/rsb.conf

#16 Updated by J. Wienke over 7 years ago

There is also a rsb.conf inside the project root folder. If you launch your binaries with PWD being the project root folder, this config is evaluated.

#17 Updated by S. Wrede over 7 years ago

Johannes: This is true. However, I suspect that Suchit wants to run the unit tests from within Eclipse. The rsb.conf in the project's root folder is defined for use with the ant-based unit tests, isn't it?

So, either he configures his spread daemon to use the ports defined in this rsb.conf or the tests from within Eclipse will fail.

#18 Updated by S. Sharma over 7 years ago

Thanks, the problem was due to presence of rsb.conf in ~/.config/rsb.conf as well as the project folder, after commenting both , socket connection was made.
I was also able to remove certain exceptions due which tests were failing, but there was a conversion exception:

*rsb.converter.NoSuchConverterException:No conveter with key rsb.protocol.introspection.HelloType$Hello$Builder registered in ConverterMap *

this seems to occur when event.setdata() is used for Hello's builder.
Do we need to register the hello builder before using?

#19 Updated by S. Wrede over 7 years ago

You are right. As you use the converter subsystem as any regular RSB application, you have to tell the framework how to convert these types. I've just added the fundamental introspection types (feel free to extend / correct if necessary) at the respective place:

https://code.cor-lab.de/projects/rsb/repository/rsb-java/diff?utf8=%E2%9C%93&rev=a78aaf9e1e6af47659d92c41cc8fb52c2515d2a2&rev_to=dfdff019f6c3f4fcd516ac506352499c41af4ef5

Please pull these changes and try again...

#20 Updated by S. Sharma over 7 years ago

I have pulled the changes but still get the same error.I even tried some other things but doesn't seem to work.

#21 Updated by J. Wienke over 7 years ago

The important thing is that the converter registration needs to be done before you create the participants.

#22 Updated by S. Wrede over 7 years ago

The code should be called through the rsb.Factory. I suppose that there is a different problem.

BTW: What kind of test are you executing? The QueryHandlerTest still fails for me with the NullPointerException at the exact same location as last week.

#23 Updated by S. Sharma over 7 years ago

I had made changes to QueryHandlerTest.java and commited them earlier, in my machine I dont get the null pointer exception anymore.Could you please pull the changes and try once more.

@Johannes - Could you please explain how the converter registration has to be done.
Do I have to register the converters somewhere in QueryHandlerTest.java

#24 Updated by S. Wrede over 7 years ago

@Suchit. You must not register the converters manually for your case.

#25 Updated by S. Wrede over 7 years ago

@Suchit. Did you push your changes?

There aren't any recent commits from your side:

https://code.cor-lab.de/projects/rsb/repository/rsb-java?utf8=%E2%9C%93&rev=feature-introspection

#26 Updated by S. Wrede over 7 years ago

And, for the converter registration, please check the corresponding example (although it should not be implemented this way for this feature):

http://docs.cor-lab.de//rsb-manual/trunk/html/examples-converters.html#tutorial-converters

#27 Updated by S. Sharma over 7 years ago

I have done the changes we discussed yesterday , test case is running now. Could you please verify if its fine and also tell me what to do next.

#28 Updated by S. Sharma over 7 years ago

Test case for ProcessInfoServerTest.java is also getting passed but throws NosuchConverterException on the console.
Also I have implemented the checks if data is correct or not, but the checks to see if the real functionality of synchronyzing the clocks is working or not ,has to be impemented ,but I dont have much idea how to do that.

#29 Updated by S. Sharma about 7 years ago

I was trying call c code through JNI in rsb java , this c code gets host name and host id of the machine by calling some functions (does the same thing as cpp implementation of introspection).the JVM throws fatal error without significant logs when I try to run this .The error is due to currentHostId function in c thats for sure but I dont get an idea since my knowledge with c is limited.
To run the program I have made a simple class TestJNIString.java in rsb.introspection package it calls the native library file in rsb/native/libs folder. Right now make configuration and genrated libmyjni.so is for linux , to compile it for MAC name has to be changed in make file from libmyjni.so to libmyjni.dylib. The make file to compile c code is in "native" folder in feature-introspection of rsb project.

#30 Updated by J. Wienke about 7 years ago

I have no idea where you got the C code from, but in its contains an error that is causing the crash you see.

First of all I had to fix several java issues before I could even compile that java code. Please make sure that the code you check in compiles.

For the error: If you add the -ggdb flag to the compiler invocation in makefile, your native library will be compiled with debug symbols so that we can use the gdb debugger to find out what is going on. After recompiling the library (make clean && make) we can now start the whole java process inside gdb to get a C debugger (from the root folder of the project):

gdb --args java -cp bin/ -Djava.library.path=native/libs/ rsb.introspection.TestJNIString

After pressing r the program will run until it crashes, which happens quite soon here. After that you can enter bt which shows the following backtrace:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fb0700 (LWP 17543)]
__isoc99_fscanf (stream=0x0, format=0x7fffe5299b8a "%[^\n]") at isoc99_fscanf.c:31
31    isoc99_fscanf.c: No such file or directory.
(gdb) bt
#0  __isoc99_fscanf (stream=0x0, format=0x7fffe5299b8a "%[^\n]") at isoc99_fscanf.c:31
#1  0x00007fffe5299a3d in tryFile (filename=0x7fffe5299b5f "/etc/machine-id") at TestJNIString.c:72
#2  0x00007fffe5299995 in currentHostId () at TestJNIString.c:40
#3  0x00007fffe5299911 in Java_rsb_introspection_TestJNIString_sayHello (env=0x7ffff00071d8, thisObj=0x7ffff7faf9a8, inJNIStr=0x7ffff7faf9a0) at TestJNIString.c:18
#4  0x00007fffed011790 in ?? ()
#5  0x00007ffff7faf9a8 in ?? ()
#6  0x00007fffed0130a8 in ?? ()
#7  0x00007ffff7faf940 in ?? ()
#8  0x000000077a271f40 in ?? ()
#9  0x00007ffff7faf9a8 in ?? ()
#10 0x000000077a272340 in ?? ()
#11 0x0000000000000000 in ?? ()

This indicates that tryFile from your code tries to open /etc/machine-id which does not exist. The name of the method tryFile implies that it should actually be graceful in case a file does not exist, but the implementation in your branch is not. So this is the root cause of the crash you are experiencing. You somehow need to fix this and check for existence of the file before trying to scan through it.

Where did you get the C code from?

#31 Updated by S. Sharma about 7 years ago

Thanks , I wrote the c code myself , taking rsb cpp as reference.

#32 Updated by S. Sharma about 7 years ago

I implemented the below methods in C and am able to call them through JNI in Java:
currentProcessId();
currentHostId();
currentHostname();
currentProgramName();
currentCommandlineArguments();
currentProcessStartTime();

the c implementaion is in rsb/native/introspectionJNI.c , can be called by test java class rsb.introspection.IntrospectionJNI.java which works fine , same methods are also called in HostInfo.java and ProcessInfo.java(Unit testing yet to be done) .

There were two doubts/problem:
1. currentCommandlineArguments() is this method required , its not present in introspection c++ version as well.
2. In currentProcessStartTime() in rsb/native/introspection.c there is a problem in parsing regex to read start times from /proc/stat and /proc/self/stat files . Even though regex is correct , there is some problem in matching of regex , I tried to resolve it but since I am not very familiar with C it was not resolved.

P.S. as you might know I will be leaving for next 12 days starting tommorow and will be back on 29th . I still have to unit test host and process info and then test the whole thing with CL introspection tool.

I will be in citec today and tommorow so you can call me anytime if its required.

#33 Updated by S. Sharma about 7 years ago

  • % Done changed from 40 to 80

Portable as well as default(native) version of introspection sender seem to be working now.I will get it verified by Jan.
Please suggest what I should be doing after this.

#34 Updated by S. Wrede about 7 years ago

On MacOS, the current way of getting the PID does not work.

java.lang.NumberFormatException: For input string: "" 
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:504)
    at java.lang.Integer.parseInt(Integer.java:527)
    at rsb.introspection.PortableProcessInfo.getPid(PortableProcessInfo.java:94)
    at rsb.introspection.PortableProcessInfo.<init>(PortableProcessInfo.java:62)
    at rsb.introspection.IntrospectionSender.<init>(IntrospectionSender.java:88)
    at rsb.introspection.IntrospectionObserver.<init>(IntrospectionObserver.java:48)
    at rsb.Factory.enableIntrospection(Factory.java:83)
    at rsb.introspection.EnableIntrospectionTest.test(EnableIntrospectionTest.java:42)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

However, according to http://stackoverflow.com/questions/35842/how-can-a-java-program-get-its-own-process-id
the following JVM call should do on most implementations:

Using the java management and monitoring API (java.lang.management):
ManagementFactory.getRuntimeMXBean().getName();
returns something like:
28906@localhost
where 28906 is the PID of JVM's process, which is in fact the PID of my app.

#35 Updated by S. Wrede about 7 years ago

Next: Command line does not work with current linux-based implementation on MacOS:

java.lang.StringIndexOutOfBoundsException: String index out of range: -1
    at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:872)
    at java.lang.StringBuilder.substring(StringBuilder.java:72)
    at java.lang.AbstractStringBuilder.subSequence(AbstractStringBuilder.java:849)
    at java.lang.StringBuilder.subSequence(StringBuilder.java:72)
    at rsb.introspection.PortableProcessInfo.getProgramName(PortableProcessInfo.java:125)
    at rsb.introspection.PortableProcessInfo.<init>(PortableProcessInfo.java:64)
    at rsb.introspection.IntrospectionSender.<init>(IntrospectionSender.java:88)
    at rsb.introspection.IntrospectionObserver.<init>(IntrospectionObserver.java:48)
    at rsb.Factory.enableIntrospection(Factory.java:83)
    at rsb.introspection.EnableIntrospectionTest.test(EnableIntrospectionTest.java:42)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

#36 Updated by S. Wrede about 7 years ago

Missing bits and pieces:

  • MacOS non-JNI implementations
  • Kind of participant not set
  • Hierarchical participants
  • Introspection Listener / Informer objects shouldn't introspect themselves...
  • Testing, Testing, Testing, ... (also load testing)

#37 Updated by J. Moringen about 7 years ago

  • Category changed from Java to Introspection

#38 Updated by J. Moringen about 7 years ago

  • Assignee changed from S. Sharma to S. Wrede

#39 Updated by J. Moringen about 7 years ago

  • Category changed from Introspection to Java

#40 Updated by S. Wrede almost 7 years ago

  • Status changed from In Progress to Feedback
  • Assignee changed from S. Wrede to J. Wienke
  • % Done changed from 80 to 100

Initial implementation done and available in wip-introspection. Complete refactoring and partial rewrite of feature-introspection. Incorporated Johannes review feedback. Tested so far on MacOS 10.10 and Ubuntu 14.04 with Java 7. Should partially work on Windows as well. That said, the introspection code definitely requires more testing.

I am handing this over to Johannes for a final check and to eventually push this to master.

#41 Updated by J. Wienke almost 7 years ago

  • Status changed from Feedback to Resolved

Also available in: Atom PDF