Archive for the ‘Miscellaneous’ Category

How to Compile and Run Open Whisper Systems’ Signal Server (TextSecure Server) (Part 2)

August 9, 2017

Last time, when I try to activate Signal Server, I got this:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
null

So, it exited the loop, and only giving a “null” as an error message. This is the most cryptic error I’ve ever get in debugging experieces.

Checking the exact source of exception reveals it is java.lang.NullPointerException.

So, the call stack can be easily obtained by breaking into NullPointerException in the java sources:

And I have:

Let’s perform of analysis of this class:

APNSender apnSender = new APNSender(accountsManager, config.getApnConfiguration());

private static PrivateKey initializePrivateKey(String pemKey)

Let’s see the content of pemKey:

lkajsdf

The content of pemKey appears to be obtained from:

Since I just create the same dummy variables for the above configuration variables, let’s differentiate it, and I have that pemKey is initialized using pushKey configuration item:

PEMReader reader = new PEMReader(new InputStreamReader(new ByteArrayInputStream(pemKey.getBytes())));

I assumed that to get the pushKey from Apple required some dollar involvement, so, let’s just create a dummy PEM Key just to make this class happy.

import org.bouncycastle.openssl.PEMReader

So, this is part of the utility created by bouncycastle, let’s get some information regarding this first. OK, so this is a PEM Certificate, let’s try to generate it:

C:\OpenSSL-Win32\bin>openssl genrsa -out rsa_1024_priv.pem 1024
Generating RSA private key, 1024 bit long modulus
………++++++
…..++++++
e is 65537 (0x10001)

Run the server, and I have:

D:\Projects\AndroidProgram\Signal\Server>java -Xdebug -Xrunjdwp:transport=dt_soc
ket,server=y,address=8880,suspend=y -jar target\TextSecureServer-1.65.jar serve
r config/textsecure.yml
Listening for transport dt_socket at address: 8880
—–END RSA PRIVATE KEY not found

No matter what I try to fiddle with the key, I got this same error message. I believe the format I’ve passed to the configuration is OK. But let’s see whether it is generated some exception again.

This time, I got java.io.IOException. Repeat the above procedure, I have:

But when I try to trap the IOException, there’s an interesting IOException just BEFORE this exception. So, let’s check it out first:

\C:\Program Files (x86)\Java\jdk1.8.0_131\jre\lib\sunrsasign.jar

Seems, it has some relation with RSA signing. From the findjar website, I’ve found that is already in rt.jar so, I think I’ll stop pursue this issue for the moment.

Let’s come back to the above rather persistent error message to understand what it try to process from the given private key.

Retrieve bouncycastle module sources from:

http://central.maven.org/maven2/org/bouncycastle/bcprov-jdk16/1.46

Which is the dependency currently used by my TextSecureServer java program at this moment.

But NetBeans complains as follows:

Not able to submit breakpoint LineBreakpoint PemReader.java : 54, reason: Line number information is missing in the class file org.bouncycastle.util.io.pem.PemReader.

Let’s verify:

This is the one that has line number information:

D:\temp\dw\io\dropwizard>javap -l application.class
Compiled from “Application.java”
public abstract class io.dropwizard.Application {
protected io.dropwizard.Application();
LineNumberTable:
line 25: 0
line 27: 4
line 28: 11

And indeed, the given class in the jar does not have line number information:

D:\temp\bc\org\bouncycastle\openssl>javap -l pemreader.class
public class org.bouncycastle.openssl.PEMReader extends org.bouncycastle.util.io
.pem.PemReader {
public org.bouncycastle.openssl.PEMReader(java.io.Reader);

public org.bouncycastle.openssl.PEMReader(java.io.Reader, org.bouncycastle.ope
nssl.PasswordFinder);

Try to find compatible environment for BouncyCastle’s version 1.46 compilation proves to be difficult. After perform some research, I decided to compile the version 1.57 instead.

D:\Projects\JavaProgram\BouncyCastle>gradlew build -x lint
Error: Could not find or load main class org.gradle.wrapper.GradleWrapperMain

This can be resolved by copying existing gradle folder in the usable projects into this project folder.

Now I have:

D:\Projects\JavaProgram\BouncyCastle>gradlew build -x lint
Starting a Gradle Daemon, 1 incompatible and 1 stopped Daemons could not be reus
ed, use –status for details
0% CONFIGURING [42s]
> root project > Resolve dependencies :classpath > gradle-cobertura-plugin-2.2.

It failed to download the above package. So again, using proxy will alleviate the problem.

But the result of version 1.57 is somewhat vague, which has many JARs that is not compatible with the intended version.

So, I have to resort to create a customized gradle build system from older 1.46 sources in D:\Projects\JavaProgram\BouncyCastle2, and after a successful compile, I verify it that it is indeed it has line number information in it:

D:\temp>javap -l pemreader.class
Compiled from “PemReader.java”
public class org.bouncycastle.util.io.pem.PemReader extends java.io.BufferedRead
er {
public org.bouncycastle.util.io.pem.PemReader(java.io.Reader);
LineNumberTable:
line 19: 0
line 20: 5
LocalVariableTable:
Start Length Slot Name Signature
0 6 0 this Lorg/bouncycastle/util/io/pem/PemReader;
0 6 1 reader Ljava/io/Reader;

So, now I try to replace the 1.46 version in my local maven repository and perform the recompiling.

But to my surprises, I found many errors.

This is strange, because the first time I try to compile it is a success. Why the second compilation process is not ?

First, I suspect maybe it is the cause of my replacing of BouncyCastle 1.46, so I revert back to the old jar and try to re-compile with exact same error message:


[ERROR] /D:/Projects/AndroidProgram/Signal/Server/src/main/java/java/util/function/LongUnaryOperator.java:[64,5] default methods are not supported in -source 1.7
(use -source 8 or higher to enable default methods)
[ERROR] /D:/Projects/AndroidProgram/Signal/Server/src/main/java/java/util/function/LongUnaryOperator.java:[66,25] lambda expressions are not supported in -source 1.7
(use -source 8 or higher to enable lambda expressions)
[ERROR] /D:/Projects/AndroidProgram/Signal/Server/src/main/java/java/util/function/LongUnaryOperator.java:[92,38] static interface methods are not supported in -source 1.7
(use -source 8 or higher to enable static interface methods)

Try to change the version as suggested by the error message, i.e change to 1.8 proves to be generated more cryptic errors. So I revert back to the old configuration, and I decided to compare the original sources to the one I try to re-compile, and I found mysterious java folder inside the sources I try to re-compile.

And suddenly I realized it is my mistake this time, because I accidentally use this source folder as a NetBeans debugging, and added unnecessary sources into the folder.

Next time I should place the debugging source files into another folder. After removing the offending folders, and try to recompile:

[INFO] ———————
[INFO] BUILD SUCCESS
[INFO] ———————

It is OK with the version 1.46 of BouncyCastle utility. So, let’s replace it now with one with line number information and see what happens when I try to re-compile, again it is a success.

To create a separate source, I created a new folder at D:\Projects\NetBeans\SignalServer to host the new location of NetBeans project.

After some studying of NetBeans to try to separate the sources for debugging and the actual program so that the above compiling error can be avoided, I realized, it is very difficult to achieve that condition.

Later on, I found that using “Remote Attach” to add new Source Root will solve the problem:

The Java and any other JARs that required source code debugging then can be placed into D:\Projects\NetBeans\Debug\src. Now, let’s start to analyze the BouncyCastle error.

First, I try to perform breakpoint at the PemReader’s loadObject, and here is the result:

The readLine function suddenly returning with null, so it end up with the —–END RSA PRIVATE KEY not found error. This indicated that the buffer is not received any of the data. Hmm, fishy.

After checking the readPemObject function of PemReader.java, I realized that the passed string should contain Carriage Return and Line Feed. But how to encode it into YAML object ?

Try to pass \r and \n proves to be futile, because readLine inside this function returns one line:

After some study of YAML using this format will solve the issue:

Now I have:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65.
jar server config/textsecure.yml
org/eclipse/jetty/alpn/ALPN$Provider

The cause of the above exit is java.lang.NoClassDefFoundError. Try to perform break point, I have:

at java.lang.NoClassDefFoundError.(NoClassDefFoundError.java:59)
at io.netty.handler.ssl.JdkAlpnApplicationProtocolNegotiator$1.(JdkAlpnApplicationProtocolNegotiator.java:26)
at io.netty.handler.ssl.JdkAlpnApplicationProtocolNegotiator.(JdkAlpnApplicationProtocolNegotiator.java:24)
at io.netty.handler.ssl.JdkSslContext.toNegotiator(JdkSslContext.java:285)
at io.netty.handler.ssl.JdkSslClientContext.(JdkSslClientContext.java:261)
at io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:751)
at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:418)
at com.relayrides.pushy.apns.ApnsClientBuilder.build(ApnsClientBuilder.java:408)
at org.whispersystems.textsecuregcm.push.RetryingApnsClient.(RetryingApnsClient.java:63)
at org.whispersystems.textsecuregcm.push.APNSender.(APNSender.java:61)
at org.whispersystems.textsecuregcm.WhisperServerService.run(WhisperServerService.java:176)
at org.whispersystems.textsecuregcm.WhisperServerService.run(WhisperServerService.java:111)
at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85)
at io.dropwizard.cli.Cli.run(Cli.java:74)
at io.dropwizard.Application.run(Application.java:89)
at org.whispersystems.textsecuregcm.WhisperServerService.main(WhisperServerService.java:276)

The above exception is tripped by the statement:

if (!JdkAlpnSslEngine.isAvailable()) in JdkAlpnApplicationProtocolNegotiator.java, which is caused by un-initialized JdkAlpnSslEngine

But why it has connection to the above org/eclipse/jetty/alpn/ALPN$Provider ? Checking the source code of JdkAlpnSslEngine.java, it is indeed there’s reference to this class:

import org.eclipse.jetty.alpn.ALPN

Performing checking to the binary JARs also confirm the non-existent of this class.

Some statement from the web regarding this issue in https://stackoverflow.com/questions/39856972/http-2-java-8-jetty-and-alpn:

Jetty’s ALPN boot jar works with both OpenJDK and Oracle’s JDK (which is based on OpenJDK).

Jetty’s ALPN boot jar must be in the boot classpath, not the regular classpath, like the documentation you linked says.

As such, you must not declare it as a dependency in your pom.xml files (there is no need to, like there is no need for you to specify a dependency on the JDK classes).

JDK 9 will have ALPN support native, there is already some work in that direction.

For java version java version “1.8.0_131” requires 8.1.11.v20170118.

Configuring command line to:

java -Xbootclasspath/p:target\alpn-boot-8.1.11.v20170118.jar -jar target\TextSecureServer-1.65.jar server config/textsecure.yml

D:\Projects\AndroidProgram\Signal\Server>java -Xbootclasspath/p:target\alpn-boot
-8.1.11.v20170118.jar -jar target\TextSecureServer-1.65.jar server config/textse
cure.yml

Will solve this issue, but now the server is continuous loop. But seems there’s no information, logs or anything to indicate what is it that is doing. Using ProcMon to find out interesting files created reveals:

D:\tmp\textsecureshserver.log

Inside this file is a wealth of information regarding the server’s activities. Hmm, good job Open Whisper Systems. Although there’s some issue should be fixed, for example a more friendly error message and some initial information of where to search vital information such as log, etc.

But I think this is the way how open source community works.

After a while, the server stops by itself:

D:\Projects\AndroidProgram\Signal\Server>java -Xbootclasspath/p:target\alpn-boot
-8.1.11.v20170118.jar -jar target\TextSecureServer-1.65.jar server config/textse
cure.yml
Multiple exceptions

Hmm, interesting, but I will stop for the moment.

How to Compile and Run Open Whisper Systems’ Signal Server (TextSecure Server)

August 7, 2017

The readme.md content from https://github.com/WhisperSystems/Signal-Server do not give a slightest hint of how to build the source.

Reading some hints in http://debabhishek.com/writes/Installing-and-Running-TextSecure-Signal-Server-on-Windows/. It turns out it should be compiled using Maven.

So, let’s download maven first. Then perform some sanity check:

C:\Windows\System32>mvn -version
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-04T02:39:0
6+07:00)
Maven home: C:\Maven\bin\..
Java version: 1.8.0_131, vendor: Oracle Corporation
Java home: C:\PROGRA~2\Java\jdk1.8.0_131\jre
Default locale: en_US, platform encoding: Cp1252
OS name: “windows 7”, version: “6.1”, arch: “x86”, family: “windows”

Now, let’s compile:

D:\Projects\AndroidProgram\Signal\Server>mvn clean install -DskipTests
[INFO] Scanning for projects…
[WARNING] The project org.whispersystems.textsecure:TextSecureServer:jar:1.65 us
es prerequisites which is only intended for maven-plugin projects but not for no
n maven-plugin projects. For such purposes you should use the maven-enforcer-plu
gin. See https://maven.apache.org/enforcer/enforcer-rules/requireMavenVersion.ht
ml
[INFO]
[INFO] ————————————————————————
[INFO] Building TextSecureServer 1.65
[INFO] ————————————————————————
Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven
-source-plugin/2.2.1/maven-source-plugin-2.2.1.pom
[INFO] ————————————————————————
[INFO] BUILD FAILURE
[INFO] ————————————————————————
[INFO] Total time: 22.536 s
[INFO] Finished at: 2017-08-03T09:51:24+07:00
[INFO] Final Memory: 7M/17M
[INFO] ————————————————————————
[ERROR] Plugin org.apache.maven.plugins:maven-source-plugin:2.2.1 or one of its
dependencies could not be resolved: Failed to read artifact descriptor for org.a
pache.maven.plugins:maven-source-plugin:jar:2.2.1: Could not transfer artifact o
rg.apache.maven.plugins:maven-source-plugin:pom:2.2.1 from/to central (https://r
epo.maven.apache.org/maven2): Connect to repo.maven.apache.org:443 [repo.maven.a
pache.org/151.101.40.215] failed: Connection timed out: connect -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please rea
d the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResoluti
onException

D:\Projects\AndroidProgram\Signal\Server>

So, perform changes in the Maven’s settings.xml:

After some lengthy POM and JARs download, I have:

[INFO] ————————————————————————
[INFO] BUILD SUCCESS
[INFO] ————————————————————————
[INFO] Total time: 24:57 min
[INFO] Finished at: 2017-08-03T11:06:09+07:00
[INFO] Final Memory: 30M/104M
[INFO] ————————————————————————

Let’s run it:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65.jar

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65.
jar
usage: java -jar TextSecureServer-1.65.jar
[-h] [-v]
{server,check,directory,vacuum,trim,stats,rmuser,accountdb,messagedb}

positional arguments:
{server,check,directory,vacuum,trim,stats,rmuser,accountdb,messagedb}
available commands

optional arguments:
-h, –help show this help message and exit
-v, –version show the application version and exit

Let’s run it using the configuration given from the above website:

java -jar target\TextSecureServer-1.65.jar server config/textsecure.yml

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
config/textsecure.yml has an error:
* Malformed YAML at line: 8, column: 3; while scanning a simple key
in ‘reader’, line 8, column 1:
+33756796138 #fake
^
could not find expected ‘:’
in ‘reader’, line 9, column 1:
localDomain: foo.org
^

at [Source: java.io.FileInputStream@9175d8; line: 7, column: 2]

So, let’s debug it:

java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8880,suspend=y -jar target\TextSecureServer-1.65.jar server config/textsecure.yml

The command line result:

D:\Projects\AndroidProgram\Signal\Server>java -Xdebug -Xrunjdwp:transport=dt_soc
ket,server=y,address=8880,suspend=y -jar target\TextSecureServer-1.65.jar serve
r config/textsecure.yml
Listening for transport dt_socket at address: 8888
config/textsecure.yml has an error:
* Malformed YAML at line: 8, column: 3; while scanning a simple key
in ‘reader’, line 8, column 1:
+33756796138 #fake
^
could not find expected ‘:’
in ‘reader’, line 9, column 1:
localDomain: foo.org
^
at [Source: java.io.FileInputStream@f76a6; line: 7, column: 2]

There’s no more helpful information. So, let’s find at what routine or module that give the above error, i.e. the Malformed YAML error string. This is done by extracting files in the JAR and perform binary search into the files by using utilities, such as WinHex. It takes some time, but I think this is one of possible starting point.

But the search turns out … nothing, either by using ASCII or UNICODE format. Hmm, so what to do ?

Try to do googling regarding the error message, I realized that the error location could be somewhere at YamlConfigurationFactory class of io.dropwizard java class utility.

Let’s obtain this source, but first I have to consult the maven repository of the version being used in this java application, and it turns out using v1.1.0.

Try to obtain the source from http://central.maven.org/maven2/io/dropwizard/dropwizard-configuration/1.1.0/dropwizard-configuration-1.1.0-sources.jar proves to be diffcult because what I got is a connection reset error.

After the source code is obtained, and placed it in proper folder on the Netbeans project, I place the breakpoint to this routine:

It is a hit, but the Netbeans is hang when I try to view the call stack.

What I only got is some information like this:

But this should be enough. Upon restarting the Netbeans, turns out to be ok.

After importing proper source code and perform some step traces, I have:

Where the location of the error is at BaseConfigurationFactory.java in io.dropwizard class:

The reason that I do not find “Malformed YAML” error message is because error string is generated programmatically, where as when I try to search for string Malformed I find many instances in the binary files.

So, let’s examine more detail of this error, which is actually originated from MarkedYAMLException:

Checking further:

This is because the config I copied from given by above website is indeed malformed, the spaces is missing, and after fixing the TextSecure.yml config file:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
config/textsecure.yml has an error:
* Unrecognized field at: push.host
Did you mean?:
– queueSize

No matter what I modify in the file, I always get this constant message error, and there’s a mysterious suggestion of queueSize. So let’s fire our binary search again shall we ?

Binary search is suggested at BaseConfigurationFactory.java and the suggestion is retrieved using e.getKnownPropertyIds().stream().

OK now, let’s enter this property using blank value and remove the unknown property, and we have:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
config/textsecure.yml has an error:
* Unrecognized field at: websocket
Did you mean?:
– webSocket
– cache
– redphone
– server
– metrics
[16 more]

To retrieve complete properties, I have to resort to debugger which is:

s3,cache,federation,read_database,webSocket,testDevices, twilio, directory, gcm, httpClient, push, database, redphone, apn, metrics, messageStore, turn, logging, server

The above keyword should be the complete keyword for the root property configuration options. There’s typo in the given sample text secure configuration from the above website.

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
config/textsecure.yml has an error:
* Unrecognized field at: webSocket.enabled
Did you mean?:
– requestLog

At this point, I started to wonder the validity of the given configuration from the above website.

After browsing some of the codings, then I realized, that the purpose of the parser is to try to map the text in to the class, at the root class is named WhisperServerConfiguration.java.

After more fiddling with the TextSecure.yml, I finally get:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
config/textsecure.yml has the following errors:
* apn may not be null
* gcm may not be null
* turn may not be null
* webSocket.requestLog may not be null

Let’s first tackle the webSocket configuration issue by passing requestLog: abc, since I don’t know the purpose that option yet, and I have:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
config/textsecure.yml has an error:
* Failed to parse configuration at: webSocket.requestLog; Can not construct in
stance of io.dropwizard.request.logging.LogbackAccessRequestLogFactory: no Strin
g-argument constructor/factory method to deserialize from String value (‘abc’)
at [Source: N/A; line: -1, column: -1] (through reference chain: org.whispersys
tems.textsecuregcm.WhisperServerConfiguration[“webSocket”]->org.whispersystems.w
ebsocket.configuration.WebSocketConfiguration[“requestLog”])

After some head scratching and source code review, I realized, this must be configuration item from io.dropwizard utility and I found that using this:

Will stop the complain. But what if I purposely mistyped the type ? In this case, I have:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
config/textsecure.yml has an error:
* Failed to parse configuration at: webSocket.requestLog.appenders; Could not
resolve type id ‘consolex’ into a subtype of [simple type, class io.dropwizard.l
ogging.AppenderFactory]: known type ids
= [AppenderFactory, console, file, papertrail, syslog]
at [Source: N/A; line: -1, column: -1] (through reference chain: org.whispersys
tems.textsecuregcm.WhisperServerConfiguration[“webSocket”]->org.whispersystems.w
ebsocket.configuration.WebSocketConfiguration[“requestLog”]->io.dropwizard.reque
st.logging.LogbackAccessRequestLogFactory[“appenders”])

Hmm, interesting. There are indeed other types of requestLog option, but let’s revert it back to console, now I have:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
config/textsecure.yml has the following errors:
* apn may not be null
* gcm may not be null
* turn may not be null

After proper settings of dummy TextSecure.yml file I have:

D:\Projects\AndroidProgram\Signal\Server>java -jar target\TextSecureServer-1.65
.jar server config/textsecure.yml
null

No exceptions, anything whatsoever, just “null”. Well, I will stop here for a moment.

Bootstrap Dialog Table Look Up Example

May 10, 2017

Here is the table look up example in action. When the page is properly setup it will show one button as follows:

When the button is clicked it will show the look up dialog box:

The data is retrieved from the server using AJAX based on user action, such as a click on table header, etc. The table is actually the extension of the original jQuery.DataTable.js open source program.

The table can be dynamically built using SELECTOR keyword:

The programmer only responsible for performing the codes inside the server side ASPX handler for given selector keyword, such as table header definition, paging request and display data. This is included in the sample NewInquiry.aspx page.

User can hover to the table and perform selection by clicking on to the table, and it will show the alert box:

This sample program is created using ASP.NET, so if you are interested to see how this page in action, there are several pre-requisite as follows:

1. The data source is using SQL Server 2005
2. Windows Server 2003 or higher with Framework 4

Sample data is included in the “data” folder on the source package.

Source files and supporting executable as follows:

https://github.com/surya-rakanta/BootstrapLookUp.git

How to Decompile MultiCharts SEF File

March 21, 2017

This article will explore whether it is possible to perform decompilation of MultiCharts’ *.SEF file back to its original Power Language Script, a variant of Easy Language Script. This is one of trading applications just like TradeStation.

The *.SEF file is created by using the “Export Read Only” menu in MultiCharts Power Language Editor (PLEdit.exe). At first glance, the content of generated file consist some header information and the actual executable file.

Let’s find whether the editor is keeping the Intermediate code, or hopefully an encrypted source code just like TradeStation’s *.ELD file.

This is done by creating an increasing more complex script and see the changes inside the *.SEF file. So, after several steps of creating, compiling and examining the file content, I realized that the PLEdit program is in fact, do not try to preserve the intermediate code, let alone the encrypted source code.

The only information is the actual executable file that is located just after the header portion of the file which typically like this:

There’s a MZ signature that denotes that portion of the block to the end of *.SEF file is actually the executable file. Using the Hex Editor I can move out the block into separate file and try to dissasemble it:

This executable file is actually a form of a DLL file because it has some exported functions, one of which is called “Create”. When I try to execute the compiled script in MultiCharts program by using the “Insert Study …” menu, it will try to load the generated i_myTesting.dll.

The myTesting is the is the name that I declared when creating the script of type Indicator (hence the i_ prefix). This is evident by viewing the Load DLL output from WinDBG program:

When “Insert Study…” is executed for myTesting script, the program will call the DLL’s Create exported function.

So, I landed on a rather hopeless case of machine code that only the computer’s CPU and MultiCharts program can execute faithfully. At this point, I try to googling out for this issue. I found that certain website does actually stated that they can decompile manually from the machine code to the script source code.

I try to communicate with one of the support of this service, and get a positive response, that indeed, yes, they can decompile and it require about 3 or 4 weeks and some compensation.

How can they do that ? This let me pause and think, maybe I do not perform a more in depth analysis. Then I have another activities to perform, but the question remains in my head. It’s like they can provide something from nothing.

So, I decided to take a more close look, by try to identify how the PLEditor generate it’s DLL file from the script. I created the most simple Indicator script called myTesting and it’s content as follows:

Then I realized that the PLEditor.exe, actually compile it’s file by using the process called StudyServer.exe.

By examining the modules loaded by StudyServer.exe, I’ve found the curious file called GccWrapServer.dll. By examining more closely, it is revealed that it is actually a COM object and it has a methods called Compile and SetCompilerSettings method. This is evident by viewing its type library using OLE View program:

From the above picture, I can see that the Compile method accepts the CPP file name and generated DLL file name. By setting the breakpoint into the Compile method, what I get from these two parameter as follows:

From the above execution code, I can see that the ecx register contains the CPP file name:

Folder content as follows:

After the Compile finishes, the *.cpp file is removed, so I try to obtained it while the program is suspended by the debugger and the content of CPP file as follows:

There are many information revealed by this generated CPP and its corresponding *.o file, one is the parameter information. So the above Create export function calls can be identified as follows:

The above rather hopeless code starts to have some meaning, while it is just a start, then the decompilation of MultiChart’s *.SEF file suddenly become possible. Much works still to be done, it’s hard, but it is indeed possible 🙂

Open Office’s Open XML SDK Saving Issue

September 9, 2016

In one of the projects that involved using the Open XML SDK for manipulating the spreadsheet files, I stumbles across some strange issue, i.e. some of the modified worksheet cell items is not get saved into the document.

I’ve spend considerable amount time to investigate this, which involved obtaining the existing source of SDK, resolves compilation issue and examine the save routine.

By examining the workings of save method inside the SDK, I figured out, that although I’ve called PutXDocument and examine the XML result inside the sheets*.xml the data is already saved, but actually there’s another layer inside the Worksheet object that still contains old worksheet data.

So, when the Close method of SpreadsheetDocument is called, it is the layer of original worksheet data that get saved into the OpenXmlPart stream, not the stream that I already modified.

Try to call Worksheet.Save method proves to be useless, because the Close method is actually use the same method when try to save all Open Xml Parts.

So, how to trigger the data flow from the modified Xml data layer back to Worksheet object ? You can use Worksheet.Reload method. This call will cause your modified Xml part flows back into the Worksheet object.

So, at this time, when you try to call the Close method of the SpreadsheetDocument, your XML layer already get saved into the Worksheet object and your modified data inside the Worksheet object will take effect.

Resolving XAOS Post Compilation Issues

September 9, 2016

When I try to load the *.xpf file containing the user formula, my compiled application refuses to process it:

zp01

Seems that the previous complaint from configuration process at the compile time should be taken seriously.

Let’s now handle the importing of the GSL library. GSL stands for GNU Science Library, and it is in the form of source code and should be compiled first to obtain the library specific to the OS of the target machine.

zp02

Examining the content of the above include files that causes a problem reveal that it is created as symbolic link statement to the actual file. There is a statement inside makefile.am file configuration inside D:\Projects\Fractal\gsl\gsl folder as follows:

zp03

In configure file:

zp04

MSYS can’t handle the symlink, so the solution is revise the above script to use cp (copy command) instead of ln -s. Make sure it does so in the config.log file:

zp05

Now GSL source will continue to compile until it stops at this stage:

zp06

This is because:

zp07

Some stage of previous compilation process is not executed due to some unknown problem, just remove *.lo and recompile inside utils folder:

zp08

Now GNU GSL Library will compile just file. Let’s move on to re-compile XAOS source:

zp09

There is a problem inside the configuration file inside the XAOS source file that causes GSL_CONFIG to be blank when the script try to check and include the GSL library, so modify to:

zp10

Depending on the location of GSL library installation. Now we have:

zp11

Copy the include file from GNU gsl folder into src\include of the XAOS source code and also the corresponding libgsl.a and libgslcblas.a to C:\mingw\lib folder.

Open XAOS program and try to load *.xpf file that has formula in it and the program crashes 🙂

Further examination reveals that I’ve used the wrong GSL library for compiling GSL with XAOS. This is verified by creating the small program and examine the data, and it turns out the result is garbage, although it compiles right.

The compiled static library libgsl.a should be taken from .libs result in the GSL directory, not from lib folder.

Re-compile XAOS after the correct library is copied.

Now, user formula runs just fine 🙂

XAOS Compilation on Windows OS

September 2, 2016

XAOS application is used to display fractal graphics, and because I want to perform close examination of the way it process the given user formula, I try to compile from the source on windows OS.

This article describe the compilation process and resolves the common issues found in compilation process on windows OS.

xa01

Since it is still a warning stage, I decided to leave it for a while and proceed to the next phase of compilation process.

xa02

The file libfilter.a is actually generated from the compiler, seems it failed to manifest at compile phase previous to linking process.

xa03

It failed to manifest because there is an error above. Download the appropriate libiconv library from MINGW site.

xa04

This is because of the problems of missing libiconv that get passed at configuration process. Remove all object files and intermediate result of compilation process, and perform re-configure.

This time, the compilation is progressing without any incident.

xa05

How to Resolve ClosedXML Error

August 24, 2016

One of my projects requires capability to create and download pre-formatted excel file to the web client. In the process of finding the suitable application packages that will handle this task, I’ve found ClosedXML utility that seems to fulfill this demand. It is also an open source application which has some advantage of cost reduction 🙂

So, I decided to perform some assessment of this software before deploy it to production stage. I’ve created my template excel file, save it in the Open XML format.

After passing several feature testing, I stop at an obscure error trying to perform copy method of the excel worksheet:

ox01

The error says “Sequence contains no matching element”. To isolate the problem, the routine is moved to desktop application. After that I obtain and compile the source code of ClosedXML application.

The debugging result as follows:

ox02

The application crashes when trying to enumerate existing ranges called “_Regression_Int” and the Ranges object throws exception because it points to non existent physical range. I don’t recall creates this weird range name, it must be an artifact of previous version of office application or there are some flaws in name creation routine.

The Open XML format is actually consist of compressed files, so I decided to decompress and examine the Workbook.xml inside the folder as follows:

ox03

There’s a range definition but there are no actual range object, so I decided to remove this pesky thing, and perform compression again. This time, the worksheet can be copied without any error.

The Sims FreePlay’s Way of Text Rendering

February 6, 2016

Gamers that plays the Sims FreePlay Android application will be familiar with the start up screen below:

sf01

Here it will show the constant message “LOADING…” and the varying message inside the blue box, which different each time when you start the application. Seems that this message is generated using some sort of random mechanism.

The actual string lies inside thesimsfreeplay_android.sb depending on the language setting of the device, and it is governed by STRING_LOADING_TICKER_XX records where XX denotes the sequence ranging from 00 to 21.

So, after the random module generates the sequence number, say 11, actually this is for “Earn big XP bonuses for reaching relationship milestones” above. This string will be retrieved and displayed on the GL Surface.

The text itself is rendered dynamically using glTexSubImage2D and it is placed on empty space of size 1024×64 texel inside existing texture channel. Each unique character to be rendered is generated on the existing texture memory as follows:

sf02

Because of the the texture memory is in inverted form, so to you can either flip the monitor to see the character or use the vertical rotation to view each of the character as follows:

sf03

Then, it is a matter of placing the right vertex position for each character to be rendered on to GL Surface.

The Sims FreePlay’s SBIN File Format

January 1, 2016

One of the data structures used in Sims FreePlay android game application is files with *.sb extention.

If you try to open it in the hex editor, you will find that it is started with SBIN signature, followed by other blocks of data, started with with other signature, such as STRU, FIEL, ENUM, OHDR, DATA, etc.

Apparently the 4 bytes signature is the marker for the block of data, and close examination will reveal that each block has 3 common data type which I will call the signature, block length and the checksum.

Let’s take the STRU block below for some illustration. This is taken from one of the *.sb file called locales.sb as follows:

sb01

From the above picture, each block contains 3 (three) data item, each of 4 bytes size. The first one (red box) is the signature, here the 0x53545255 is STRU, the second one (blue box) is size of the data inside of this block in bytes, here the size is 0x0C, the third (green box) is some kind of checksum value of the data block.

If you try to change just one byte of the data portion, the checksum value will be different and it is up to the game developer here what to do when this condition occurs, for example, it will only show blank data for some text information.

By reading the signatures block, you may get some idea what data that is contained in the file. It should contain structure, fields, field names, and the actual data.

So, how to describe each block into structures, fields, header name and data ?

Let’s examine the data portion of STRU block as follows:

sb02

The size of each STRU data is 6 (six) bytes, and from the above example it contains 2 (two) structures, and apparently contain 3 (three) data types, each has the size of 2 (two) bytes.

Take the first structure information (red box), the second item is used to determine the location of the field and the third item is total field number in the structure.

So, for the first structure (red box), the field location is 0x0000, this is multiplied by 0x8 which is the size of field data block to get the pointer position if first field data. From the information of third data item, the first structure has 2 fields, i.e. 0x0002.

The field information of second structure (blue box) is located at offset 0x0002 times 0x8 which is 0x10, and it has 0x0009 or 9 fields.

To determine the pointer location of fields information, just add the offset with the starting of the data portion of the FIEL block which is just after 4 bytes checksum. This is illustrated by the picture below:

sb03

For the above sample, the start of field data block is located at offset 0x38, and each has the size of 8 (eight) bytes as denoted by red box. This is the first field of first structure.

To determine the first field of second structure, from the above calculation, the offset 0x38 should be added with 0x10 and we arrive at:

sb04

As denotes by blue box, this is the first field of second structure. And the field information contains the 2 bytes field sequence followed by another 2 bytes that denotes the field type. The field type will determine how the structure reader obtained the data for each field.

Take the first field inside first structure, the field sequence number is 0x0002. This sequence is related to the data portion inside CHDR structure, which is used to determine the field name.

Below is the picture of CHDR structure:

sb05

The 0x43484452 (CHDR) actually the signature, it has 0x198 bytes length and 0x4A35B08C is checksum value for the data block. After the check some, here comes the CHDR detail structure, which each has 8 (eight) bytes.

From the above picture, the red box is for sequence zero, blue box is sequence 1 (one) and so on. You can see that sequence 0 (zero) contains no information. The first information actually resides on second record (blue box) which is sequence 1.

Each CHDR detail structure consists of 4 bytes that denotes offset location of the field name inside CDAT structure, and 4 bytes that denotes field name length.

Take the offset location of sequence 1 which is 0x00000001. This value is added to the start of data portion of CDAT to get the name of the sequence.

Below is picture of CDAT structure:

sb06

From the above sample, the 0x43444154 (CDAT) is signature, followed by 4 bytes length (0x00000195) and 4 bytes checksum (0x4FD442C6), so we arrive at offset 0x384 as the start position of field names information. This is added by 0x01 obtained from CHDR structure, and we arrived at offset 0x385, and by the 0x08 bytes as its length, we get the field name information (red box) which is “Language”.

So, what’s the name of field sequence 0x02 and 0x03 ? The answer is “languageid” and “localeid”. I will leave the task to derive this conclusion for the reader 🙂

The combination of bytes inside OHDR and DATA block is used to locate the detail records for each table. Below is the part of OHDR structure:

sb07

The value 0x4F484452 is OHDR signature, it has 0x0C length, and just after the check sum, there are 4 bytes (0x00000001) which denotes the header type. This value is used to calculate offset from start of DATA record to get first detail table information. If the type 0x00, the offset is 2 bytes, 0x01, the offset is 4 bytes, 0x02 is 8 bytes.

Below is the picture of DATA structure:

sb08

As usual, the value 0x44415441 is DATA signature. The start of data information is located at offset 0xB4 and because OHDR header type is 0x01, it is added 0x4 bytes to get the location of first table information block, as denoted by the blue box above.

The table information block has similar structure to the field information block, so you can see that bytes at offset 0xB8 to 0xB9 (0x0B) is table name sequence number which is “locales” table.

The 4 bytes at offset 0xBC to 0xBF (0x00000C) is the offset location to get the table id for this table. This value is added to the start of DATA structure which is 0xB4 to get 0xC0.

Below is byte information at address location 0xC0:

sb09

The blue box is the table sequence number which will be used to locate the first data block or record of the related table, for this example is first record of locales table.

The red box denotes the next table information block, which has 0x0E as the sequence number which is “languages” table. Again, offset value 0x00000018 is added to 0xB4 to get 0xCC which is 0x00000002 or the table sequence number for “languages” table.

The table sequence number is then multiplied by 4 to get the location of table record detail. So for id 0x01 we get 0x04. This value is added to the start of OHDR data to obtain information about the size of detail records for the table in question:

sb10

From the above picture, 0x4F484452 is OHDR, and you already seen that 4 bytes started at offset 0x9C is the start of header information block. So the calculated 0x4 is added to the offset 0x9C, and we arrived at 0xA0 (blue box).

It has 0xE2, and to get the location of first record data, this value should be divided by 0x8, so we get 0x1C. The value 0x1C is added to the start of DATA block offset (0xB4) so we get 0xD0:

sb11

For each block of data record, it has 0x8 bytes prefix (red box) which contains information about the data record block such as total number of record such as 0x00000B, at offset 0x4 thru 0x7. The actual data starts at offset 0xD8.

To get the length of each record, we should consult the offset information for the last field of the table:

sb12

As you can see from the picture above, offset information is located at 0x8C thru 0x8D, so we get 0x0010. This should be added to the data size of the last field that could be 0x2, 0x4 or 0x8 bytes, depending on the field type information located at offset 0x8A thru 0x8C (0x000C). In this case, the size is 0x02 so the actual record length is 0x12.

Here is some information about the field type and its size of last byte:

0x03,0x04,0x0C,0x0D,0x14,0x15 -> last byte size = 0x2

0x05,0x06,0x0A,0x0F,0x11,0x12,0x13,0x16 -> last byte size = 0x4

0x07,0x08,0x0B -> last byte size = 0x8

The string type (0x0D) inside the detail record is treated as the same with methods to find the field name, which is in the form of sequence number, for example, let’s view the first detail record for “locales” table:

sb13

For the first field data information 0x000F, which is localeid, it is treated as a sequence number and referenced using CHDR structure to get string data from CDAT structure, in this case, 0x000F is “en”.

With all of the relevant information is revealed, it is possible to manually traverse the detail record for this format, or creates some program to do this task 🙂