Debugging ssh-keygen dsa on Solaris8

[Scott Burch]

Andy Polyakov wrote:
> 
> > > 2) If I use 0.9.5a of openssl I do not have any of these problems with
> > > the exact same compiler and options. (This is why I think some code
> > > change is causing this problem.
> >
> > I do not see any significant code changes in the dsa-code between
> > 0.9.5a and 0.9.6b...
> > Please change line 69 in openssl-0.9.6b/crypto/dsa/dsa_lib.c from
> >   static DSA_METHOD *default_DSA_method;
> > to
> >   static DSA_METHOD *default_DSA_method = NULL;
> 
> Oh come on... 'static's are zeroed at startup. But Lutz is right about
> default_DSA_method being the prime suspect for being unaligned. I would
> suggest to setup watchpoint on default_DSA_method and see who's
> modifying it. If you can't do it yourself make executable available for
> download.
> 
> Andy.


I have some good news to report:

1) Doing the following worked:

Please change line 69 in openssl-0.9.6b/crypto/dsa/dsa_lib.c from
> >   static DSA_METHOD *default_DSA_method;
> > to
> >   static DSA_METHOD *default_DSA_method = NULL;

I compiled with -g -xarch=v8 for both openssl 0.9.6b and openssh 2.9p2
with Sun Forte Developer Update 2 and everything worked (with the above
change)! I will be testing -xarch=v9 and 64 bit compilation to see if
that works as well. In any case thanks for the suggestions. I still have
to test the applications, but I will report back on that soon. Just to
be sure I used a clean source tree and tried to compile with the above
options and DSA was broken. If you need anything else from me at this
point let me know. I have already started to build a clean jumpstart
image to test with...what I would like to ultimately determine is why
some people have been able to compile things without going through all
this trouble...I'd hate to see everyone else go through all this
testing, hopefully I can help prevent that.

Thanks again,
Scott