This is your resource to discuss support topics with your peers, and learn from each other.
05-07-2009 06:32 AM - edited 05-07-2009 08:35 AM
In my application I encounter strange behavior of network layer. Sometimes my application receives time-out exceptions however requests still reaches the server after some period of time from several minutes to servarl hours with average about half an hour. I don't know exact conditions when and why this happens in production but I can reproduce it using voice call. The issue wih voice call is reproducible both on real device and on simulator. Moreover, it seems that the issue is not in my application. Using some tricks I can reproduce it even on standard BlackBerry Browser at least in simulator.
Excuse me in advance for very long and boring post but this is the only way I see to state the facts I know.
I'm going to get into standard BlackBerry browser implementation as reference implementation of network usage. It is hard to do, since RIM doesn’t give us code but it is still possible to do something. To follow my experiment you will need some JDE, BlackBerry device simulator, MDS, some network sniffer and some patience. I used JDE 4.2.1 with BB-8800, MDS from JDE 4.1 and MS Network Monitor (available for free at
The way I use to discover RIM’s implementation of different things is debugging. When JDE stops at breakpoint it shows several nice things (accessible via View menu): Call Stack (Alt+7) and Locals (Alt+4) for simple things and VM Byte Code, VM Locals and VM Stack for some advanced debugging (this time we’ll need it). And it works even if debugger stops not in ours code! All we need is to find public methods that RIM uses, place breakpoint there and wait until it will be triggered by RIM’s code.
Let’s try to do it. To begin with I need two breakpoints: in Connector.open and inside RadioInfo.isDataServiceSuspended. I used a simple application that calls both methods to set breakpoints there (see further). Application is quite simple. There is a single screen with two menu items that just call the methods I need.
package test;
import net.rim.device.api.system.RadioInfo;
import net.rim.device.api.ui.*;
import net.rim.device.api.ui.component.LabelField;
import net.rim.device.api.ui.container.MainScreen;
import javax.microedition.io.Connector;
import javax.microedition.io.HttpConnection;
import java.io.IOException;
/**
*/
public class RimApp extends UiApplication
{
private static class MyMainScreen extends MainScreen
{
public MyMainScreen()
{
super();
add(new LabelField("Debug Test"));
addMenuItem(new MenuItem("Connector.open", 1, 1)
{
public void run()
{
doConnectorOpen();
}
});
addMenuItem(new MenuItem("RadioInfo.isDataServiceSuspended", 2, 2)
{
public void run()
{
boolean res = RadioInfo.isDataServiceSuspended();
}
});
}
private void doConnectorOpen()
{
//Do net request in a separate thread
new Thread(new Runnable()
{
public void run()
{
HttpConnection conn = null;
try
{
String url = "" + ";deviceside=true";
conn = (HttpConnection) Connector.open(url, Connector.READ_WRITE, true);
conn.setRequestMethod(HttpConnection.GET);
conn.getResponseCode();
}
catch (Exception e)
{
System.err.println(e);
}
finally
{
try
{
if (conn != null)
conn.close();
}
catch (IOException e)
{
System.err.println(e);
}
}
}
}).start();
}
}
public RimApp()
{
pushScreen(new MyMainScreen());
}
public static void main(String[] args)
{
new RimApp().enterEventDispatcher();
}
}
Zipped source code with JDE workspace
First set breakpoint at
conn = (HttpConnection) Connector.open(url, Connector.READ_WRITE, true);
line and activate corresponding menu item. When debugger stops on breakpoint delete it (we will not need it further), step into (F11), set breakpoint and disable it (not delete, but just disable with Ctrl+F9, we’ll activate it later).
Again set breakpoint at RadioInfo.isDataServiceSuspended call and activate corresponding menu item. When debugger stops on breakpoint and step into. Here we want to set breakpoint not at the beginning of the method. So don’t delete our breakpoint now, you might need it if you miss proper place first time. Show Call Stack debug window (Alt+7) and Use Step into and Step Out (Shift+F11) until you reach such stack trace: CallCommandHandler.isActive(). Show VM Byte Code window and activate it. Use Step Over (F10) until you reach ireturn statement (Note: you’d better dock this particular window. Sometimes Step In/Out works strange if this window is not docked. You can see what I have in JDE at this moment on attached screen shot). Set breakpoint at ireturn, disable it and release debugger (F5). Now you can delete breakpoint in my code.
We have to disable breakpoint because these methods, especially CallCommandHandler.isActive, are used a lot by background threads and breakpoint will be triggered too often.
Now we are almost ready. Start MDS. Start your sniffer and configure it to catch HTTP traffic. (In MS Network monitor create new capture tab, set HTTP as either Display or Capture filter and Start Capture). In JDE Show Breakpoints window (Alt+F9) and enable breakpoint in Connector.open. Navigate to any URL that is not cached by the browser (to be sure I use some new fake search request in google). Debugger will stop at breakpoint at Connector.open. Find in call stack method RawDataCache.get(FetchRequest), click on it in the call stack window, and set and disable breakpoint there. Disable Connector.open breakpoint and let simulator Go. Look at the sniffer. You should see your HTTP requests.
OK, this is what I expected in this case. But interesting things happens with time-out. We’ll simulate it with voice call. But since Browser checks it, we have to cheat it. Start voice call (to any number) and Answer it on pop-up dialog. Switch back to the browser having active voice call. Enable breakpoint at RawDataCache.get(FetchRequest) and open some other not cached URL. Debugger will stop at our breakpoint. Enable breakpoint at CallCommandHandler.isActive and release debugger (F5). Debugger will stop at it. However there might calls from code different from the browser. This is not what we need. If you had one just use Go (F5). Call that you need will be on the same thread with RawDataCache.get(FetchRequest) in the stack trace. To me the easiest way to catch it is to check that start of call stack is RenderThread.run(). When you are at breakpoint with proper call stack, show VM Stack window (Alt+6). You’ll see single value 0x00000001 (Boolean true) in the stack. Change it to 0x00000000 (false), disable breakpoint and let simulator Go. Browser will show progress bar. Wait for several minutes until error message appears. Close error message. Now disconnect your voice call and look at the sniffer. You’ll see you HTTP requests there! (at least I had). If you are curious you also might set breakpoint on java.io.InterrupetedIOException and ensure that browser’s request failed with timeout.
We know that request didn’t leave device at the moment when connection was closed by timeout but still it reached the server after it. And here is my main question: Is such behavior bug or feature? To me it seems like bug. For example, in my application I send some updates to server via HTTP POST request and expect to receive update data from server. In such case server processes request, updates data but its response is ignored by mobile client because connection was closed long time ago. Moreover in case of any network error my application believes that update didn’t reach server and sends it again and again. It means that the same update is performed several times by the server. This is definitely not the thing I want (and not the thing I expect). Is there way to avoid such behavior? Is RadioInfo.isDataServiceSuspended() check before actual request enough for it? Now I check RadioInfo.getState, RadioInfo.getSignalLevel and Phone.getActiveCall before making request to avoid the issue in voice call scenario but the issue is still somehow reproducible in production.
Thanks in advance, any advice will be appreciated.
05-09-2009 07:22 AM
05-09-2009 10:04 AM
I tried to read your post,. but I decided to wait until the DVD comes out.
05-10-2009 08:04 AM - edited 05-10-2009 08:12 AM
OK, I'll try to explain the issue that I have in simpler way. Still my description is long but this time it doesn't take sniffer, debugging and other shamanic stuff.
My application does some client-server communication using HTTP POST requests. User might perform some update that is sent to the server and server updates DB. Sometimes my application receives time-out exception at conn.getResponseCode() call, but it occurs that the update reaches the server anyway. Moreover usually it reaches the server after something like 30 minutes after request. The bad thing is that the server has performed update but the client thinks that the server didn't.
What makes things much worse is that one of key features of application is that some updates might be done "offline". User might do action when there is no connection to the server. Mobile client updates local copy of data and puts corresponding update request to "output queue". There is background thread that tries to send update from the "queue" until success. Of course, time-out is not treated as success and request is re-sent again and again. And if this is that twisted kind of time-out, then after long period of time the whole batch of the same update requests reaches the server and update is performed several times for one user action.
I don't know when and why this happens in production. I was able to reproduce this issue using voice call in non-3G networks. I mean networks where voice call prohibits data transfer. I started voice call, switched to my application and perform action, waited for 20 minutes (much more than connection timeout) and terminated the call. Results: time-out in my application and update performed on the server after those 20 minutes. Using simulator and sniffer I found that request didn't leave device until the end of the voice call. It means request is sent to the network when my application has already received timeout! (Note: if you have any application that does client-server communication you can try to reproduce this issue in the same way.) When I found it, I added check of RadioInfo.getState, RadioInfo.getSignalLevel and Phone.getActiveCall before making actual request. Anyway the issue is still somehow reproducible in production.
For a long time I thought that the issue is somewhere in my code but now it is not so clear to me. In my first post I described the way how to reproduce the same issue using simulator and standard BlackBerry browser. The issue is not reproducible with BlackBerry browser without cheating because they use RadioInfo.isDataServiceSuspended() check. Now I also use checks mentioned above and the issue is not reproducible in my application in voice call scenario as well. But still I see that it is somehow reproducible.
So my questions are: Has anybody else seen such issue? Is the issue in my code or in RIM's code? How to fix/workaround the issue? (For example: is RIM's RadioInfo.isDataServiceSuspended() check enough?)
Thanks for patience to read another long and boring post,
Serg
05-14-2009 01:07 PM - edited 05-14-2009 01:13 PM
Following some advices
I decided to create a video where I reproduce steps I described in my first post.
For more detailed description of my issue see my previous post. In short the issue I see is that under some unknown conditions my connection receives time-out but after big period of time request reaches the server and is processed by the server. In this video I reproduce the same issue using standard BB browser. The most important thing that actually makes me think that there is some issue in the OS implementation is that request didn’t leave device until end of voice call when caller application has already received time-out error. And my questions are: Has anybody else seen such issue? Is the issue in my code or in RIM's code? How to fix/workaround the behavior?
Video takes about 8 minutes. I prepared video in several formats:
Source code of my test application that I use in the video is available at
Thank you for your attention. Hope this time I'll get some reasonable advice.
05-15-2009 01:14 PM
We are potentially seeing a similar issue. In our case we are using interface=wifi to force wifi connectivity. In the event log at the time of the behaviour we are seeing
net.rim.tcp OPsx -7 events, which I believe corresponds to a timeout.
However, traffic for other apps seem to be working at this time.
Is there error events in the event log at the time this is happening for you?
05-15-2009 04:20 PM - edited 05-15-2009 04:32 PM
Cadmium, thank you for reply. Unfortunately I have no "real" event logs. Still, from your description I think that you have different issue. First of all we don't use WiFi at all. Also I believe that in our scenario no application can access the network (If you are interested, see details further in this post).
The real issue is reproducible only in production and relatively rarely. However we already have enough customers to see it every day and usually several times a day. The only reason why we found this issue was strange behavior of our server-side. When we looked at server logs, we found that sometimes batch of the same requests arrives to the server (because of our "output queue" mechanism, see my second post) so we started tracking it. We was able to reproduce the issue with voice call and added corresponding check to our application. And since issue is still reproducible, we know that this is not the "real-life scenario". Unfortunately, we don't know what the "real-life scenario" is.
The only thing we have is server-side logs that are not very helpful. We can see that device loses connectivity and after some interval mostly from 20 minute to 1 hour device gains it back. There also might be such strange connection behavior over shorter time interval that we don't catch because it is doesn't corrupt our DB. Anyway when connectivity is restored, device continues usual work (except for batch of the same request that were created during this "bad" time). So even if we contact our customer (that is really-really hard and can't be done very fast) when we notice new accident (that is also "not immediately"), we can't get anything interesting.
Also we can see that the intervals between requests creation is very close to the Connection timeout (by default we use different intervals). That's why we believe that the behavior is similar to what we see in the voice call scenario. I imagine situation such way (almost 100% speculation): for some reasons there is cellular network coverage (we check it), but there is no data transfer. And network layer doesn't "see" such situation to raise exception immediately. Our HTTP-request is put to the OS-level output queue but can't be sent. Our application receives timeout and closes the connection but OS-level output queue is not cleared so when connection is restored our request is sent.
05-27-2009 11:32 PM
Have you found any solution to this? I've been seeing the same problem. The short description is:
1. Some code is running in a thread that tries to do an http post to our server about once every two minutes, with some data
2. If there is no data connection available when the post is attempted, it times out, and the data is retained in a buffer for a repost attempt in the next cycle
3. Somewhere internally in the blackberry, it keeps a cache of the data. As soon as the data connection comes back, all the timed-out posts suddenly happen automatically (and completely outside of control by my app) by some hidden process outside the scope of my app.
4. My app, thinking the data was never posted, now posts yet another copy of the data.
Because of this, 2 problems occur:
1) The same data gets posted multiple times (sometimes many many times, depending on how long the data outage was)
2) In extreme cases (such as when the user has been on an airplane), the buffer (wherever it is) seems to fill up and somehow fail, and after that *nothing* gets posted from my app, even after the data connection becomes available again.
The quick fix that I've implemented is to wrap the posting routine with:
if (RadioInfo.isDataServiceOperational()) { ..... }
05-31-2009 08:39 AM
Idonen, there is something similar in your description to what I have. I know that it sounds cruel, but it's nice to hear that I'm not the only one. Can you provide some more details? How often and under what conditions your issue is reproducible? And I didn't understand your second issue. If nothing is send what is the issue? My issue is reproducible rarely and under unknown conditions. Plain out of coverage is not enough. However it might be so because of checks that I use and I've been using them from very-very old builds. In current production build I already use RadioInfo.getState, RadioInfo.getSignalLevel (these two are used since the very beginning) and Phone.getActiveCall. The issue is still somehow reproducible, rarely but often enough to occur every day among people who use our application. In current beta build I use RadioInfo.isDataServiceOperational() but I don't have enough statistics to say whether it eliminates the issue. Currently I don't know any better solution that suits me. And AFAICS native BB browser doesn't have better solution (see my previous posts). I have one idea but it is unacceptable to me now: you can use some site that you believe is always alive (microsoft.com, google.com or something) and do HTTP GET request to it before doing actual upload request.
08-25-2011 08:09 AM - edited 08-25-2011 08:15 AM
I have exactly the same problem - the only difference is that it occurs everytime when I get a network timeout!
The steps to reproduce the issue are the following:
1. Disabling Data Services under Preferences->Mobil Network
2. Sending a HTTP request from inside my App. This results in a timeout and an error message in the GUI.
3. Switching to the System Settings and enabling Data services under Preferences->Mobil Network (app is still opened in background!)
4. Switch back to the app. The last request (that had a timeout before) is sent now immediatly !
This is really annoying because the GUI informs the user about a fail but in reality the request has been transmitted successfully!
Is it possible to disable this kind of output cache/queue ?
I am wondering because this thread seems to be very old and I couldn't find any other posts with users that are running into the same problem.