revework.htm: The flexlm challenge and cooperative reversers' work

The flexlm challenge and cooperative reversers' work

flexlm
Courtesy of fravia's pages of reverse engineering
september 1999

Many an interested reader has written to me and asked for a description of the "way" we work. Wannabie crackers, studiosi of the reversing world, lone wizards and even potential "employers" seem to believe that there is a precise answer to this question.
Alas, It is not so: each cracker, each reverser has a different working style. Team group work is seldom applied, but when it is it gives great fruits. Unfortunately reversers are for their nature rather "individualistic" oriented, and the percentage of people that "asks and wants" is out of any sound proportion with the percentage of those answering and giving... a result, most probably, of the awful and "greed-oriented" society we are compelled to live in.
However, due to the simply staggering amount of mail I have received in this matter, and as my only hope to dam the flood of inquiry, I have decided to publish here an example of a "team work" (sort of) that has been taking place last week on my meassageboard between some reversers that are -rightly- seen as the mightiest FlexLM-reversing experts on this planet. Those of you that will read what follows will be able to understand what it means, in sheer terms of reversing-power, when good reversers work together.

This is quite a long document, very important for all those interested into reversing group work, not only for FlexLM reversing purposes. I personally find this kind of work (that, remember, is done absolutely for free and for knowledge's sake) fascinating. A beautiful reading.

Reversers involved
Dan ~ VoxQuietis ~ Pilgrim ~ Nolan Blender ~ Q

Reversers at work: the FlexLM thread
September 1999

Dan to group (this is the email that started the whole thread)

FYI flexlm v6.1 and lc_new_job()

I have done some more research and on a target that
used flex 6.1. If you have a target that uses this
version, and calls lc_new_job instead of lc_init,
then you will not get the correct encryption seeds
by using the current methods of retreiving the seeds.

If you search the lmgr library in v6.1 you will see that
l_svk is not called anymore in lc_init or lc_checkout.
Instead, l_sg is called. As I stated in another message, l_sg
behaves differently when called in lc_init and lc_checkout.
This happens when the target calls lc_new_job instead of
lc_init. If you read the docs, lc_new_job provides enhanced
security over lc_init. This is a description of some of
what that security is.

There is a flag that is got to from the "job" structure. ("job"
is flexlm's version of global variables - the structure is passed
to and fondled by a lot of functions). This
flag tells l_sg to either behave like l_svk (the old seed routine)
or to call a new function. The flag also indicates a valid address
is in a vector that l_sg will call. Here is a high level flow
of the process:

call lc_new_job {
call ? (l_2bufg maybe) (sets up decryption vector)
call lc_init (l_sg flag not set yet) {
call l_sg {
call l_key
return old style seeds
}
}
set l_sg flag in job struct
}
...
call lc_checkout {
...
call l_sg {
flag set so call decryption vector (l_8indexes in lm_new.obj) {
generate random string by calling time()
store string in job structure
decrypt keys and xor with some of the random bytes
}
}
...
use seeds and random bytes to authenticate the license entry
compare authenticator to one in license entry
...
}

Note lc_new_job is wrapped around lc_init and always sets this l_sg flag.
Another thing to note is the alternate seed generator is not in the library
but in a separate object file lm_new.obj. If you read the documents about
lc_new_job, they make a point that you have to link lm_new.obj if you call
lc_new_job. This threw me off because I guess my target linker located the functions
in lm_new.obj far from the core of flexlm.

An implication of not including lm_new.obj is that maybe globetrotter will change
this module even for the same version of flex. Another reason I suspect this
is that there is a constant in lm_new.obj that my target and the flex distribution
differed by. Maybe this is related to the "patch level". Also, I didn't find
lc_new_job in the .dll which would mean its only supported by static link.

If you run across a target using this, I would sugest breaking on calls to l_sg.
The first call it will not go to the lm_new vector. Subsequent calls to l_sg will go to
lm_new vector. When l_sg returns from the vector it will have decoded the seeds
and xored them with a random 32-bit value. That value is stored in the job structure.
For my target, the random bytes were at offsets 0xb (msb) 0x8 0x13 and 0x9 (lsb) of the
job structure. Combine these 4 bytes into a 32-bit value to xor with the hidden
seeds that are in the vendorcode structure. You still need to call l_svk to get
vendorkey5 or get it from the first call to l_sg.

dan

VoxQuietis to group

Hi Dan,

I am currently struggling through a v6.1 protected target, which incorporates the
lc_new_job routine. As I learned from your post I need to observe the output of
seed generation routine both during lc_init (i.e. when lc_init is started by
lc_new_job) and after that (i.e. when lc_checkout is processing the checksum seeds).
My problem is that I both can't locate lm_new.obj file and I have some problems in
finding the point where the additional seed scrambling information stored /
applied to the encryption seeds.

Thus it would be great, if You could give me the
code snippet, where you observed the
modified seed regeneration.

With respect to the missing lm_new.obj I am somewhat confused, since I have
the understanding, that it would be required for the build of a v6.1 demo application.
The obj-file would allow to generate a test application with known seed code,
which would greatly ease the reverse engineering of the seed code hiding process.

Bye then (and thank you in advance)
VoxQuietis

Dan to group

VoxQuietis,

Finding l_sg in the target:
Easy to find in lc_init. Search for the default seed
check i.e 87654321 and 12345678. The call just before this
should be l_sg.

Finding the vendorcode5 in target (pc version):
Trace through l_sg when called. The first time
will be via lc_init. First it checks that the job
parameter is nonzero. Then you will see that it checks a
pointer in the job parameter, I think offset +53.
So something like mov eax, [esi+53]. Then if that
pointer is nonzero, it will dereference the pointer
to check a bit in offset +141. So maybe a
test [eax+00000141], 80. The code will fail at one
of these points the first time called and will
fall through to the rest of l_sg. The final result
of this is that it will overwrite the vc.data with
the vc.data^vendorcode5. This is where you can recover
vendorcode5. This is the general flow - maybe modified
in your target.

Finding the lm_new routine in target:
Several ways to do this -
Break on calls to l_sg. The first time should be from
lc_init. After that it should be from lc_checkout.
You could also set breaks on lc_init and lc_checkout to
verify the flow of the program. Trace l_sg from the
call in lc_checkout. It should pass the above test
of bit at +141, 80. If it doesn't pass that test, then
your target did not call lc_new job. If it does pass,
it will call the lm_new routine.
Another way to find this routine (dead code) is to locate
time() in your target. Then look for a routine that calls this
a bunch of times in short sucession. Look for a divide
instruction (IDQ?) Can't remember the nemonic. This should
follow the calls to time(). I would recommend
the tracing approach.

Description of the lm_new routine:
1. Random number generation -> job structure
Checks if job parameter is valid. If not, calls
memset and sets up a fake random structure with
I think 0xcc. However job parameter should be
valid. Therefore it will call time() (return in
eax version). It calls this interleaved with
xors and shifts of constants.
2. Generate "checksum" over vendor name.
Uses divide instruction (IDQ?). Don't be confused
its just doing two mods. One is a mod 4. The other
is a mod (strlen(vendorname)). The strlen was called
somewhere in this routine. Tracing this routine will
make it apparent what its doing.
3. Combine above stuff with vc.data and vc.key[1,2]
Handles vc.data[0] first, then vc.data[1]. Uses the
same bytes of random data for both but in an "inobvious"
manner. Will store seeds^random back in vc.data.
4. subtract 5 from some of the random data.
Not sure why.

Regarding missing lm_new.obj:
This is strange because my distribution had lm_new.obj.
I believe you though because on another platform I couln't
find lm_new object either. I think you will find lm_new
code with the above approach, though.

Tell the message bord how it goes...

dan

Pilgrim to group

Yo dan, Vox!

Excellent work.
Are you going to post all this to fravia as a tutorial when it's finished?
I'm busy looking at FlexLm 6.1 too.
Seems globetrotter are learning from fravias pages ;-)

Vox, with reference to lm_new
Looking at the FlexLm 6.1 SDK, for Windoze,
in C:\Program Files\flexlm\v6.1\i86_n3
have a look at the makefile, we see lm_new.obj is made then deleted as part of the make:

lm_new.obj : ..\machind\lsvendor.c lmrand2.obj ..\machind\lm_code.h
lmrand1 -i ..\machind\lsvendor.c
$(CC) $(CFLAGS) $(INCS) /c lmcode.c
$(LD) $(LFLAGS) /out:lmrand2.exe lmcode.obj lmrand2.obj \
$(LIBS) $(CLIBS)
del lm_new.c
lmrand2 -o lm_new.c
$(CC) $(CFLAGS) $(INCS) /c lm_new.c

If I read this correctly it:

compiles lmcode.c to give lmcode.obj
links lmrand2.obj and the new lmcode.obj to give lmrand2.exe
deletes any old lm_new.c
runs lmrand2 to generate new random values in lm_new.c
then compiles lm_new.c to give lm_new.obj

Looking at lm_new.c we see all the time() calls referred to by dan.

With regards finding the routines in the target, I used the FLIRT facility in IDA.
Make your own sig file from the static library, lmgras.lib ( this has a reference to l_sg )
Dissasemble your target, run the signature and you've got all the functions labelled for you.

Keep up the good work,

pilgrim

Dan to group

pilgrim,

I could write a tutorial - I just didn't know if it would
be redundant because I don't know what has been submitted
to fravia's site. I'll write one anyway and sumbit it.

I was going to look more at lmrand1,2. I remember building
and seeing some funny stuff going on. I just didn't investigate
it. That makes more sense than my original theory of globtrotter
giving out different copies of lm_new.obj. I think one possible
way to break this is to exploit lm_new.c. I know that if you
zero out the random string that the function will give you the
right seeds if you have the right "magic" for the checksum. I
think the only thing a cracker would need would
be the constant from the target, put it into a modified lm_new.c,
along with vendorcode and then generate their seeds.
I also think that vendorcode5 is not needed in this case.
I haven't tested it, but I think vc5 could be set to 0 and you
would get the right authenticator. I'm not sure.

With regards to IDA. I have never used it. This FLIRT feature
would have helped a lot. I will have to investigate. I used a
more primitive approach which was probably unnecessary but I learned
a lot in the process.

dan

Nolan Blender to group

I have to admit that I'm a little bit confused about what is going on here.
When I look at the basic lmclient which is supplied with the 6.1 sdk, I
don't see any of my encryption seeds or vendor keys in the executable, so
so far so good. It appears as though lp_checkout calls lc_new_job, which in turn calls the
routine in lm_new. lc_init is later called, however at that time the
corrected data (xored seeds +first 4 vendor codes) are available for examination.

It seems that if you can locate the real lc_init call which is located in
lc_new_job, then you should be able to locate
the required data.
The futzing about that is done
in lm_new appears to be done in order to
prevent wholesale scanning of the executables
in order to extract keys.

Dan to group

nb,

I do not know the details of building a client and have not
persued that path. I have not studied the globetrotter supplied client.
I know that that would be good to do.

It is strange for me have this much information about a target's protection.
APIs, libraries etc.
My target is not necessarily flexlm, but the program that is using
flexlm. There is a subtle difference.

I have studied my target. I know where lc_new_job is called
in my target. It is not called within a checkout routine.
Here is the high level flow of flexlm related calls:

lc_new_job(???) lc_set_attr(job,0x38,"license path")
lc_checkout(feature a...)
lc_checkout(feature b...)
lc_checkout(feature c...)

After reading your reply, I looked at the lm_new routine some more.
If you pass as the first parameter 0, this routine will return the
uncovered seeds without xoring them with random data. However if
the first parameter is job, the routine will return with the
seeds xored with random data. The random data will be stored
in the job at offsets +0xb +0x8 +0x13 and +0x9. Before this
call, those offsets are zero.

Your client must be passing zero as the first parameter to this routine -
if not then something is wrong.

If your client is lmcheckout.c all I see in there is CHECKOUT which
is a macro that calls lp_checkout. This is not the same as my client.
A disassembly of lp_checkout would be interesting...or check of first
parameter to lm_new

When it is called in my target, job is the first parameter. Therefore it xors
with random data. However in both cases, it probably later xors with
those offsets in job because in your case they are zero and in my case
they are the values needed to uncover the seeds.

I have not researched all the calls in the api. But looking at it I don't
see lp_checkout. I see lc_checkout. I would assume people developing would
not use lp_checkout. I don't think my target did.

dan

Q to group

It's really much easier than all that. lc_init still gets called even if lc_new_job is present. And, the call to lc_init will have the real vendorcodes. At that point, just use l_svk as always. All lc_new_job does is keep the vendorcodes from showing up as global data.

VoxQuietis to group

Dear Q,

You are definetely right with respect to the
vendor codes. But there remains a little
problem (if I understood correctly what's
going on) and that's the encryption seeds.
These are not needed for lc_init, and the
Globetrotter guys decided to garble them in
order to annoy us.
To me it seems that there are two ways off
retrieving the seeds: First one could investigate
the garbling mechanism. Dan did that, as he
described the double word that is used to Xor
the seeds. Yet the general problem seems to
be unsolved (maybe Dan did it?): How to modify
the Flexlm routines so that they spit out the
real encryption seeds.
A second approach might be to examine the check
of the security string. Remember that Acme told
us, that Flexlm (once) did compare the actual
string from the license file with the expected
one. Hence it should be possible to find out
the differences between the new lc_checkout and
the old one. I briefly scanned that, but it
looks like being a pain due to lengthy spaghetti
code. But I think it is a possible approach.

Best regards
VoxQuietis

Dan to group

Q, Vox,

After reading these replys, I am starting to understand
something. I guess people are building their own clients
from the flexlm sdk and reverse engineering them.

I admit I don't understand the build process of a client.
I have not looked at this so I don't really know how
the seeds are stored in the executable image. I did
not persue that path.

My target was a real application -
not something fabricated by globetrotter. From
what nb says, the default client does some of the
new security by hiding the seeds differently. But
the default client does not appear to do the random
part. This random part basically assures the seeds
are never "in the clear" while the program is executing.

As far as "modifying" the Flexlm routines. As I said
before one way to have the lm_new routine give you the
clear seeds is to zero out its first parameter. If its
first parameter is already zero, then you don't have
a client that uses the "random" security. If it is nonzero
it should point to a "job" structure. Zero out the pointer
on the stack, or where ever it is ($o0 on sparc) and the
routine will give you the seeds.

If you still don't understand what I'm talking about then
thats fine too. Maybe you won't run across a client that
does this. If you never have a problem retrieving the seeds
without using my advice then just ignore what I said.
I did run across the problem and so I'm reporting it.

As far as scanning the authenticator generation, I went
down that path when I was cracking this. I think for
MY client it was in good_lic_key(). It went something like this:

good_lic_key { <- can't remember exact name
...
l_sg() <- here is where seeds are uncovered AND randomized
extract_date()
l_ckout_crypt() <- can't remember exact name
if (l_ckout_crypt failed) jump to error -8
...
}

This l_sg call will call the lm_new routine.
The "l_ckout_crypt" was where the seeds from l_sg were used to generate
and compare the authenticator. good_lic_key was called from somewhere
in the path of lc_checkout. Names might not be right due to me forgetting.

As I said before this is MY targets behavior. I would not be surprised
if flexlm behaved differently for YOUR client.

dan

VoxQuietis to group

Dear Dan,

first of all my compliments to Your great work on Flexlm v6.1. I followed the discussion
between You and Nolan from the beginning, having a certain feeling, that You were working
in a direction, that would help me in my actual license generation problem. I'm now
working on that for a couple of weeks, and first I thought, there were some troubles with
some attributes I didn't set correctly while using lc_crypt / lmcrypt.
But then I suddenly realized, that I have been misleaded by the randomization trick
of Globetrotter, that You first described.
Once I read the Flexlm v6.0 documentation very carefully, yet I didn't do that with the
v6.1. Indeed I downloaded the v6.1 SDK just a couple of days ago, being convinced that
v6.0 would be sufficient to generate working licenses (I think that still, but I am no
longer fully convinced on that).
According Your remarks I found the random bytes in the job memory. I do agree to your findings,
i.e. bytes 0xb:0x8:0x13:0x9 form a word, which leads to constand values when Xored on to the
randomized seeds.
Yet I am still failing in generating a working license for my target.
Having no other possibility to understand what's going on I might be forced to build an
app using the Flexlm SDK, in order to understand the relation between the randomized seeds and
real seeds.

I would also like to stress, that my target delivers one set of seeds for the first
lc_init call and a different set after the derandomizing. No need to say that apparently
both seeds aren't the correct ones. (When
setting the first parameter to zero I get the
seeds of the first lc_init call, the one
I figured out weeks ago)

I will continue to examine my target. It might be possible to build a demo app with the keys
and seeds I know. And finally it might be necessary to code a brute force attack,
which is feasible, since there are 32 unknown bits, only. I will post (possible) results on
this messageboard.

Bye then and best regards
VoxQuietis

The function resulting from lmrand implements two working modes:
one with randomization of the seeds and one without. Within the
disassembly this looks like the following

;head of function
:004D9FB3 55 push ebp
...
compute and store lenght of vendor string
...
:004D9FCF 8B4508 mov eax, dword ptr [ebp+08] ; lm_job memory space
...
:004D9FE6 85C0 test eax, eax
:004D9FE8 0F8485010000 je 004DA173 ; jump to clear seed code regeneration

; start of spaghetti like randomization
:004D9FEE 8D5E04 lea ebx, dword ptr [esi+04]
:004D9FF1 57 push edi
:004D9FF2 E8D93AFDFF call 004ADAD0
:004D9FF7 83C404 add esp, 00000004
...
:004DA171 EB10 jmp 004DA183

;head of clear seed regeneration
:004DA173 6A0C push 0000000C
:004DA175 8D45EC lea eax, dword ptr [ebp-14]
:004DA178 6A00 push 00000000
:004DA17A 50 push eax
:004DA17B E8D04CFDFF call 004AEE50
:004DA180 83C40C add esp, 0000000C

;head of checksum over vendor code - randomized function jumps in here
;this checksum is required for the seed recovery -> compare to lc_init!
* Referenced by a (U)nconditional or (C)onditional Jump at Address:
|:004DA171(U)
|
:004DA183 33FF xor edi, edi

* Referenced by a (U)nconditional or (C)onditional Jump at Address:
|:004DA1B1(C)
|
:004DA185 8BC7 mov eax, edi
:004DA187 99 cdq
:004DA188 F77DF8 idiv [ebp-08] ; modulo operations (described by Dan)
...
now follows the regneration of the seeds (with or without randomization)

Interesting question is the role of the magic numbers corresponding
the seed recovery. I changed magic number a8f38730 to 5e2f5b24, which resulted in the correct seed generation even within lc_init (just before
the comparison with 87654321 and 12345678). Maybe You guys could check,
whether this trick works for your targets, too? (Think this would speed
up the fishing of the codes for unkown targets)

Bye then
VoxQuietis

Dan to group

group,

I have a possible explanation of why there is a descrepancy
in behavior of lc_new_job. It has to do with the behavior
of lmrand2.

Try this:
use the v6 blenderd lm_code.h.
delete lm_new.obj
make lm_new.obj
now look at lm_new.c. Ok you might have done this already.
Now edit lm_code.h.
Change vkeys to "invalid" value. I changed vk1 lsb from 4 to 5.
Now delete lm_new.obj and make lm_new.obj. Now edit lm_new.c.

What does this show? lmrand2 is looking at the keys for more than
just to obfuscate them. In this case, it probably failed a checksum
so generated this code. However, I also compared the output of lmrand2
with the blenderd keys to output of lmrand using my targets keys and
there are some differences not due to "randomization".

It looks like the blenderd keys are "randomizing" the data but I haven't
compiled lm_new.c from this output.

Maybe I'm wrong - just a suggestion...

dan

Dan to group

VoxQuietis,

It is helpful to examine lm_new.c as to make
sense of these two "working modes" you describe.

When you "make lm_new.obj" with your target's lm_code.h,
you get a lm_new.c that is similar to - but not
exactly like what your target used.

This lm_new.c is compiled into what your disassembly
showed. You will indeed see the two "%" mod signs
in the c code.

The two working modes are actually called in different
places from what I understand. One function is called in
lc_new_job to form the initial vendor code structure and vname.
The other function is the one I described under l_sg.

You can actually modify lm_new.c to test out these functions.
You will see the first function "unpackages" the vc and vname.
If you then call the next function with a first param
of 0, you will see vc.data becomes your original seeds.

However, this does not show the exact magic that the target
is using, which is necessary as these functions are a
"matched pair".

Also, part of the confusion is that there are two l_n36_buf (forgot
exact name) pointers.

dan

Nolan Blender to group

I've done some experimentation and here's some code that may
be helpful in analysing how lm_new works.

First, as a baseline, here is the information for blenderd, so that
everyone knows what the secret values should be.

#define ENCRYPTION_SEED1 0xae37b151
#define ENCRYPTION_SEED2 0x6fde7999
#define VENDOR_KEY1 0xc450f9f4
#define VENDOR_KEY2 0x4d12be88
#define VENDOR_KEY3 0xf52bcf4d
#define VENDOR_KEY4 0x3309994c
#define VENDOR_KEY5 0xaefa9027
#define VENDOR_NAME "blenderd"

Link this program against lm_new.c and it will be clear what
is happening.

#include

/* Supposedly these would be secret values */

#define BLENDER_VENDOR_KEY5 0xaefa9027

#define ENCRYPTION_SEED1 0xae37b151
#define ENCRYPTION_SEED2 0x6fde7999

/* function that we get from lm_new.c */
extern int (*l_n36_buf)();

/* function pointer that will be set later. */
void (*l_n36_buff)();

void main(int argc, char *argv)
{
int retcode;
int i;
int magic_xor_value;

/*
* union for testing what we get back
*/

union
{
char charbuf[1024];
int intbuf[256];
} myunion;
char myvendorname[1024];

/*
* structure to demonstrate use if valid job passed in.
*/
struct s_tmp
{
int i;
char *cp;
unsigned char a[12];
} myjob;

/*
* Clear buffer so we know that changes are due to lm_new
*/

for (i = 0; i < 256; i++)
{
myunion.intbuf[i] = 0;
}

/*
* for this exercise, myjob really doesn't matter.
*/
myjob.i = 66;

/*
* PART 1:
* Null call initializes values inside lm_new
*/

printf ("Part 1: Null pass to decode routine\n");
retcode = (*l_n36_buf)(0, myunion.charbuf);
if (retcode != 0)
{
printf ("Problem with first call to lm_new initializer.\n");
printf ("retcode = %d\n", retcode);
return;
}

/*
* This one retrieves the secret keys
*/
retcode = (*l_n36_buf)(myvendorname, myunion.charbuf);
if (retcode != 1)
{
printf ("Problem with second call to lm_new.\n");
printf ("retcode = %d\n", retcode);
return;
}

printf ("myvendorname: %s\n", myvendorname);
for (i = 0; i < 10; i++)
{
printf ("before intbuf[%d] = %08x\n", i, myunion.intbuf[i]);
}

/*
* Test with 0 job for extracting key only
*/

(*l_n36_buff)(0, myvendorname, myunion.charbuf);

printf ("\n");
for (i = 0; i <10; i++)
{
printf ("after intbuf[%d] = %08x\n", i, myunion.intbuf[i]);
}

/* Check encryption seed 1 */
if (myunion.intbuf[1] == BLENDER_VENDOR_KEY5 ^ ENCRYPTION_SEED1)
{
printf ("Encryption seed1 OK: %08x\n", myunion.intbuf[1]);
}
else
{
printf ("Incorrect Encryption seed1 is: %08x should be: %08x\n",
myunion.intbuf[1]^BLENDER_VENDOR_KEY5, ENCRYPTION_SEED1);
}

/* Check encryption seed 2 */
if (myunion.intbuf[2] == BLENDER_VENDOR_KEY5 ^ ENCRYPTION_SEED2)
{
printf ("Encryption seed2 OK: %08x\n", myunion.intbuf[2]);
}
else
{
printf ("Incorrect Encryption seed2 is: %08x should be: %08x\n",
myunion.intbuf[2]^BLENDER_VENDOR_KEY5, ENCRYPTION_SEED2);
}

/*
* PART 2:
* Simulation of what might happen in real
* program for key recovery.
*/

retcode = (*l_n36_buf)(0, myunion.charbuf);
if (retcode != 0)
{
printf ("Problem with first call to lm_new initializer.\n");
printf ("retcode = %d\n", retcode);
return;
}

/*
* This one retrieves the secret keys
*/
retcode = (*l_n36_buf)(myvendorname, myunion.charbuf);
if (retcode != 1)
{
printf ("Problem with second call to lm_new.\n");
printf ("retcode = %d\n", retcode);
return;
}

/*
* Now call with an actual job, so highly secret data
* is stored with the job.
*/
(*l_n36_buff)(&myjob, myvendorname, myunion.charbuf);

printf ("\n");
for (i = 0; i < 10; i++)
{
printf ("(with job) after intbuf[%d] = %08x\n", i, myunion.intbuf[i]);
}

/*
* extract the xoring value for the job now.
*/

magic_xor_value = ((long)(myjob.a[5]) << 0)
| ((long)(myjob.a[0]) << 8)
| ((long)(myjob.a[1]) << 16)
| ((long)(myjob.a[4]) << 24);
printf ("Magic xor value: %08x\n", magic_xor_value);

/*
* now fix the values in the array.
*/
myunion.intbuf[0] = myunion.intbuf[0] ^ magic_xor_value;
myunion.intbuf[1] = myunion.intbuf[1] ^ magic_xor_value;

/* Check encryption seed 1 */
if (myunion.intbuf[1] == BLENDER_VENDOR_KEY5 ^ ENCRYPTION_SEED1)
{
printf ("Encryption seed1 OK: %08x\n", myunion.intbuf[1]);
}
else
{
printf ("Incorrect Encryption seed1 is: %08x should be: %08x\n",
myunion.intbuf[1]^BLENDER_VENDOR_KEY5, ENCRYPTION_SEED1);
}

/* Check encryption seed 2 */
if (myunion.intbuf[2] == BLENDER_VENDOR_KEY5 ^ ENCRYPTION_SEED2)
{
printf ("Encryption seed2 OK: %08x\n", myunion.intbuf[2]);
}
else
{
printf ("Incorrect Encryption seed2 is: %08x should be: %08x\n",
myunion.intbuf[2]^BLENDER_VENDOR_KEY5, ENCRYPTION_SEED2);
}

return;
}

I've only tested this on a unix machine, if it doesn't work on a Windows machine,
tell me.

I also looked at the routines in lm_new, and you can strip a lot of useless
junk out of that file. After all the goo was removed, this is all that
remained. I linked against this code, and everything seemed to work as
before. Perhaps this was to discourage object debuggers from figuring out
what was going on.

#include "lmclient.h"
#include
#include

/*
* Only variables that seem to do anything.
*/

static char l_var_173 = 0;
static int l_208func = 136;
static char l_buf_155 = 110;
static int l_192index = 19;
static int l_buff_203 = 80;
static char l_func_153 = 101;
static int l_registers_177 = 136;
static int l_232var = 153;
static int l_216registers = 18;
static unsigned char l_96buff = 36;
static char l_166var = 100;
static char l_counter_249 = 48;
static int l_234reg = 9;
static char l_146ctr = 98;
static int l_220var = 77;
static int l_236ctr = 51;
static char l_buff_257 = 0;
static char l_ctr_163 = 114;
static int l_196bufg = 30;
static int l_index_201 = 244;
static char l_172ctr = 0;
static int l_186indexes = 160;
static int l_buf_179 = 219;
static int l_224buff = 207;
static char l_254buf = 48;
static int l_198indexes = 97;
static char l_ctr_159 = 100;
static char l_buff_169 = 0;
static int l_ctr_189 = 64;
static int l_idx_245 = 1;
static int l_var_241 = 6;
static int l_202index = 249;
static int l_bufg_227 = 245;
static int l_idx_221 = 77;
static char l_func_161 = 101;
static int l_reg_183 = 247;
static int l_248buf = 103;
static int l_func_205 = 196;
static char l_buf_149 = 108;
static char l_counter_253 = 46;
static int l_counter_225 = 43;
static int l_counter_237 = 4;
static char l_buf_251 = 54;
static int l_228counter = 76;
static int l_212index = 190;

static
void
l_ctr_11(job, vendor_id, key)
LM_HANDLE *job;
char *vendor_id;
VENDORCODE *key;
{
unsigned long *keys;
unsigned long signature;
#define SIGSIZE 4
char sig[SIGSIZE];
unsigned long x = 0x87822e5a;
int i = SIGSIZE-1;
int len = strlen(vendor_id);
long ret = 0;
struct s_tmp { int i; char *cp; unsigned char a[12]; } *t, t2;

sig[0] = sig[1] = sig[2] = sig[3] = 0;

if (job) t = (struct s_tmp *)job;
else t = &t2;
if (job)
{
t->a[0] = (time(0) & 0xff) ^ 0x2e;
t->a[4] = (time(0) & 0xff) ^ 0x68;
t->a[5] = (time(0) & 0xff) ^ 0x0;
t->a[1] = (time(0) & 0xff) ^ 0x97;
}
else
{
memset(t2.a, 0, sizeof(t2.a));
}

for (i = 0; i < 10; i++)
{
if (sig[i%SIGSIZE] != vendor_id[i%len])
sig[i%SIGSIZE] ^= vendor_id[i%len];
}
key->data[0] ^=
(((((long)sig[0] << 3)|
((long)sig[1] << 1) |
((long)sig[2] << 0) |
((long)sig[3] << 2))
^ ((long)(t->a[5]) << 0)
^ ((long)(t->a[0]) << 8)
^ x
^ ((long)(t->a[1]) << 16)
^ ((long)(t->a[4]) << 24)
^ key->keys[1]
^ key->keys[0]) & 0xffffffff) ;
key->data[1] ^=
(((((long)sig[0] << 3)|
((long)sig[1] << 1) |
((long)sig[2] << 0) |
((long)sig[3] << 2))
^ ((long)(t->a[5]) << 0)
^ ((long)(t->a[0]) << 8)
^ x
^ ((long)(t->a[1]) << 16)
^ ((long)(t->a[4]) << 24)
^ key->keys[1]
^ key->keys[0]) & 0xffffffff);
t->cp -= 0;
}
static
int
l_4ctr(buf, v)
char *buf;
VENDORCODE *v;
{
static int l_8var;
extern void (*l_n36_buff)();
if (!buf)
{
l_8var = 0;
return 0;
}
if (l_8var >= 1) return 0;
if (!l_n36_buff) l_n36_buff = l_ctr_11;

memset(v, 0, sizeof(VENDORCODE));
v->keys[1] += (l_220var << 24);
v->data[1] += (l_ctr_189 << 0);
v->behavior_ver[3] = l_254buf;
v->data[0] += (l_reg_183 << 16);
v->keys[2] += (l_bufg_227 << 24);
v->behavior_ver[2] = l_counter_253;
buf[1] = l_buf_149;
v->type = (short)(l_counter_237 & 0xffff) ;
v->keys[0] += (l_index_201 << 0);
v->data[0] += (l_registers_177 << 0);
v->behavior_ver[4] = l_buff_257;
v->keys[3] += (l_228counter << 0);
v->keys[2] += (l_counter_225 << 16);
v->keys[0] += (l_func_205 << 24);
buf[4] = l_ctr_159;
v->data[0] += (l_186indexes << 24);
v->keys[2] += (l_224buff << 8);
v->flexlm_version = (short)(l_var_241 & 0xffff) ;
v->keys[1] += (l_212index << 8);
v->data[1] += (l_192index << 8);
v->flexlm_patch[0] = l_248buf;
v->keys[3] += (l_232var << 8);
v->data[0] += (l_buf_179 << 8);
buf[7] = l_166var;
v->keys[0] += (l_202index << 8);
buf[3] = l_buf_155;
buf[8] = l_buff_169;
v->keys[1] += (l_216registers << 16);
v->keys[1] += (l_208func << 0);
v->data[1] += (l_196bufg << 16);
buf[9] = l_172ctr;
v->keys[0] += (l_buff_203 << 16);
v->data[1] += (l_198indexes << 24);
v->keys[3] += (l_236ctr << 24);
buf[5] = l_func_161;
v->keys[2] += (l_idx_221 << 0);
v->behavior_ver[1] = l_buf_251;
v->behavior_ver[0] = l_counter_249;
buf[2] = l_func_153;
v->keys[3] += (l_234reg << 16);
buf[0] = l_146ctr;
buf[10] = l_var_173;
v->flexlm_revision = (short)(l_idx_245 & 0xffff) ;
buf[6] = l_ctr_163;
++l_8var;
return 1;
}
int (*l_n36_buf)() = l_4ctr;

Dan to group

nb,

This program worked but also illustrated some of the descrepancies
that I am talking about. One is that I did not need vendorkey5
at all to recover my seeds. And the target did not use vendorkey5
either. Another is the calling order/parameters of the two routines
in lm_new. From what I saw, the function performed to recover the
seeds did not use enough info to even have a derivative of vk5.
I am not certain.

Your routine seems to call the "decode routine" first with a null.
My target does not. It calls the decode routine like this:
lc_new_job {
char name[50];
VENDORCODE vx;
memset(&vx, 0, sizeof(VENDORCODE));
ret= l_counters_1(name, &vx); <- first call to decode routine
ret= l_counters_1(0,0); <- second call to decode routine
call lc_init
set alt encryption flag
return
}

Then to recover the seeds a call like this could be made
(setting job to 0 gives clear seeds - not seeds ^ vk5)

LM_HANDLE job;
memset(&job, 0, sizeof(LM_HANDLE));
if (random call) l_8reg(&job, name, &vx); else l_8reg((int) 0, name, &vx); <- gives clear seeds

There are exactly 3 calls to lm_new related routines - the two in lc_new_job
to the decode routine. And the one under l_sg to the decrypt routine.
There is no need at any point to use vk5.

If I am interpreting your code correctly - you are emulating how
the calls are done in your client. It seems depending on how you
use the api that it uses lm_new.c differently. I am not sure.

dan

Nolan Blender to group

The program I posted doesn't really use
vendorcode5 either - it's just there to demonstrate
correctness of the decoded keys. The calling
sequence which you describe earlier is what I
see as well - my call is at l_sg+005c where
it calls the decryption The second part of
the code is where some emulation of what happens
in the normal initialization routine is done.

The code isn't an emulation of what happens in
the flexlm clients - just an example of how the
code could be used.

The part about the invalid checksum generating
different lm_new.c files is interesting - I checked
the program with keys which had valid checksums
but were invalid for other reasons, and I got
the same result you did with the munged keys.

Your point about vendorkey5 not being required is
a good one though - I had included it to show
the correctness of the keys and to close the
loop on how the hiding works.

I didn't mention this earlier, but testprog.c
isn't meant to emulate what happens in the
flexlm based client, but rather how the routines
work, and how the correct keys can be derived if
a call with a non-null job pointer was done.

I am going to have to do some more tracing of what
goes on inside the program. The xoring with
vendorcode5 could occur before the call in l_sg
to the decoding routine.

Thank you for your assistance - the info you
have provided has been of great assistance in
figuring out how lm_new works.

I have done a detailed trace of the calling
sequences, and what you describe appears correct.
I am going out of town for a bit, but I will
post stack traces and argument values when I get
back.

later,

-NB.

You'r deep inside fravia's pages of reverse engineering, choose your way out!

red

links

+ORC

bots wars

tools

mail_fravia

Is reverse engineering illegal?