680927 - Startup crash in mozcomp.dll with Oracle Enterprise, F5 Networks, Passlogix V-GO, IBM Tivoli Access Manager SSO

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[Scoobidiver (away)]

It's #8 top browser crasher in 7.0b1.
It happens at startup.

Signature	mozcomp.dll@0xab51
UUID	321efbb0-03e6-440a-a482-2cf1d2110819
Date Processed	2011-08-19 16:08:18.495986
Uptime	6
Last Crash	17.2 hours before submission
Install Age	17.2 hours since version was first installed.
Install Time	2011-08-19 05:55:20
Product	Firefox
Version	7.0
Build ID	20110816154714
Release Channel	beta
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
CPU	x86
CPU Info	GenuineIntel family 6 model 23 stepping 10
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x0

Frame 	Module 	Signature [Expand] 	Source
0 	mozcomp.dll 	mozcomp.dll@0xab51 	
1 	mozcomp.dll 	mozcomp.dll@0xaeb2 	
2 	xul.dll 	nsDocLoader::FireOnStateChange 	uriloader/base/nsDocLoader.cpp:1339
3 	mozcomp.dll 	mozcomp.dll@0x51a3b 

More reports at:
https://crash-stats.mozilla.com/report/list?signature=mozcomp.dll%400xab51

Comment 1

[(mostly gone) XtC4UaLL [:xtc4uall]]

Strange enough seems to be Win XP SR3/Windows 7 only.

Comment 2

[Scoobidiver (away)]

(In reply to XtC4UaLL [:xtc4uall] from comment #1)
> Strange enough seems to be Win XP SR3/Windows 7 only.
And also on Win XP SP2.

With combined signatures, it's #2 top browser crasher (lot of dupes because of consecutive startup crashes).

Reports for all signatures at:
https://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A7.0&platform=windows&range_value=1&range_unit=weeks&query_search=signature&query_type=startswith&query=mozcomp.dll&build_id=&process_type=any&hang_type=any&do_query=1

Comment 3

[Scoobidiver (away)]

It is now #12 top browser crasher in 7.0b1.

7.0 correlations by module give:
    100% (21/21) vs.   0% (37/23554) mozcomp.dll
         67% (14/21) vs.   0% (14/23554) 7.0.1.151
         33% (7/21) vs.   0% (7/23554) 7.0.1.243

It seems related to IBM Tivoli Access Manager 7.0.1:
https://www-304.ibm.com/support/docview.wss?uid=swg24020069

Comment 4

[Christopher Blizzard (:blizzard)]

Contacting Kev to see if he can follow up.

Comment 5

[LegNeato]

Ping?

Comment 6

[LegNeato]

Does this happen on Firefox 6? Can we get steps to reproduce?

Comment 7

[Marcia Knous [:marcia]]

According to crash stats, this crash only happens in Firefox 7 builds. It was first seen in an 7.0a2 build and then subsequently seen in the various betas such as:

Beta 1 20110816154714
Beta 2 20110824172139
Beta 4 20110902161802

all the way up to the latest beta.

There are very few comments - will try to contact some users to see if I can some STR.

Comment 8

[Scoobidiver (away)]

(In reply to Christian Legnitto [:LegNeato] from comment #6)
> Can we get steps to reproduce?
It's a startup crash, so having IBM Tivoli Access Manager is enough.

There's one crash in 7.0a2/20110720225139, so the regression changeset belongs to:
http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=cc6e30cce8af&tochange=82f49f622e9d
There are no crashes in 6.0, so the regression changeset doesn't belong to:
http://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=d3a2732c35f1&tochange=cf0a29826586
The combination of both gives the following regression range:
bug 659468, bug 659048, bug 669487, http://hg.mozilla.org/releases/mozilla-aurora/rev/75552dfd25c2 js cset, bug 669284, bug 669975, bug 664916, bug 668699, bug 666079, bug 665964, bug 666964, bug 670452, bug 669158, bug 659468, bug 666452, bug 671053, bug 671066, bug 671422, and bug 670870.

(In reply to Marcia Knous [:marcia] from comment #7)
> will try to contact some users to see if I can some STR.
Ask them to refine the regression range using Firefox 7 Aurora builds.

Comment 9

[Marcia Knous [:marcia]]

So far only one response from a user: "All I know is that if my Microsoft Outlook is running, I can never start up Firefox 7."

Comment 10

[Benjamin Smedberg]

None of those bugs looks obviously suspicious to me. I don't think the stack trace here is going to be especially useful without symbols, so I suspect the next step is to either install the IBM software and look at this in a VM or get the relevant symbol file from IBM or get one of their engineers to give us a real backtrace.

Comment 11

[LegNeato]

Yes, let's install it and bisect. Kev, can we contact ibm? We need to resolve this ASAP

Comment 12

[Marcia Knous [:marcia]]

I tried installing it in a VM after registering for an account, but I cannot get past the signon screen without a wallet set up.

Comment 13

[Marcia Knous [:marcia]]

From another user experiencing the issue:

"Problem was that I moved my windows profile
local Mozilla settings to a new machine to try keep saved links and
passwords.

What I didn't move was the profile configuration files in Mozilla
application settings so the profile could not be found.

It is all working well again now!"

Comment 14

[LegNeato]

I poked the Access manager user group on twitter but have not heard back. marcia, does it install an add-on? If so, can we get an ID?

Comment 15

[Marcia Knous [:marcia]]

Unfortunately after installing it in a VM, I could not get past the login screen to see if anything was installed. I had to rollback my VM. 

Kev - Do we have any contacts at IBM?

Comment 16

[LegNeato]

Marcia, I mean does Firefox show an extension that we could blocklist if need be after you installed it.

Comment 17

[Marcia Knous [:marcia]]

After installing the software it made me reboot my entire system, and then I was at the sign in prompt and could go not further. I will repeat my steps again to see if I can check before the reboot occurs, but my recollection is that I had to reboot at that point to finish the installation.

(In reply to Christian Legnitto [:LegNeato] from comment #16)
> Marcia, I mean does Firefox show an extension that we could blocklist if
> need be after you installed it.

Comment 18

[LegNeato]

Ah, that sucks. You should be able to look at the filesystem directly from the host machine and see if it dropped an add-on, right?

Comment 19

[Ian Gregory]

I'm experiencing this bug but do not have IBM Tivoli Access Manager installed. I do have Oracle Enterprise Single Sign On Manager installed, and the bug doesn't manifest itself if this is not running (as soon as it is started, Firefox crashes).

Crash ID fac5d33b-3e97-4ff0-a637-42f5e2110921

Comment 20

[Scoobidiver (away)]

(In reply to Ian Gregory from comment #19)
> I do have Oracle Enterprise Single Sign On Manager installed
It exists also Tivoli Access Manager for Enterprise Single Sign-On. I wonder if these two products are based on the same code.

Comment 21

[Matthew Middleton (:zzxc)]

A user on SUMO reported that running Firefox as an administrator is a workaround to this crash.

Comment 22

[Chris]

I too have Oracle ESSO and am unable to use FireFox 7 because of this bug.

Comment 23

[Chris]

Aditional info:
FF7 still crashes when Oracle ESSO is paused, you need to completely shut-down ESSO on the workstation before you can load FireFox 7. FF6 works fine on same machine with and without ESSO running

Comment 24

[sjhwilkes]

FYI:  F5 Networks SSO exhibits the same issue too.

Comment 25

[Marcia Knous [:marcia]]

Chris: Does the workaround Matthew mentioned in Comment 21 work for you?

Comment 26

[sjhwilkes]

No neither run as Administrator, nor safe-mode make any difference.
(In reply to Marcia Knous [:marcia] from comment #25)
> Chris: Does the workaround Matthew mentioned in Comment 21 work for you?

Comment 27

[Marcia Knous [:marcia]]

This is currently the #2 top crash in early 7 data. kev - do we have a contact at Oracle we can reach out to?

Comment 28

[Stephen Fisher]

I'm using v-go Single Sign-On software, and the problem repros. Profile Manager, Safe Mode, and default startup all crash immediately on startup. When submitting crash dumps, it crashes immediately too [when restart is checked]

Shutting down v-Go SSO desktop software "resolves" the issue.

Most recent crash is at: https://crash-stats.mozilla.com/report/index/e9a88df1-598a-45be-ad45-f47e42110928

Comment 29

[Muhammed Hasan]

Another workaround has been posted in this Thread - https://support.mozilla.com/en-US/questions/880028

Comment 30

[Benjamin Smedberg]

Is there a way to reproduce this that doesn't involve having an active account somewhere? I'd like to reproduce this in a VM.

Comment 31

[Marcia Knous [:marcia]]

We have reached out to a few folks at Oracle and are waiting to hear back. I have asked them for an answer to the question that Benjamin poses in Comment 30.

Comment 32

[Chris]

(In reply to Marcia Knous [:marcia] from comment #25)
> Chris: Does the workaround Matthew mentioned in Comment 21 work for you?

I am unable to test this because I am on a locked-down XP box at work

Comment 33

[Rabs210]

Short term workaround documented here:

https://support.mozilla.com/en-US/questions/880625

Comment 34

[Marcia Knous [:marcia]]

Attachment: Workaround for the crash

Please add the empty "FileName" value to the  "HKLM\SOFTWARE\Passlogix\Extensions\AccessManager\MozHO\molten\{ec8030f7-c20a-464f-9b0e-13a3a9e97384}\4.0" key, and the crash will disappear. The file is attached.

Oracle is investigating the cause of the crash but in the meantime this should work.

Comment 35

[Ian Gregory]

Appreciate the workaround, but I would expect most users experiencing this bug are in corporate environments (like me) and thus unable to make this registry change.

Comment 36

[Scoobidiver (away)]

Can someone with one of the SSO find the regression range using Firefox 7 Aurora builds (7.0a2)?

Comment 37

[Chris]

I have just tried the registry change and it does NOT fix the issue 

OS:
OS Name	Microsoft Windows XP Professional
Version	5.1.2600 Service Pack 3 Build 2600

ESSO: v10.1.410

Comment 38

[Chris]

(In reply to Chris from comment #37)
> I have just tried the registry change and it does NOT fix the issue 
> 
> OS:
> OS Name	Microsoft Windows XP Professional
> Version	5.1.2600 Service Pack 3 Build 2600
> 
> ESSO: v10.1.410

CORRECTION:
Sorry, I created the empty key but not the "Filename" entry. I have now done so and the workaround works.

Comment 39

[MikeBY]

(In reply to Ian Gregory from comment #35)
> Appreciate the workaround, but I would expect most users experiencing this
> bug are in corporate environments (like me) and thus unable to make this
> registry change.

Agree with Ian's comment,but at least workaround can be applied by corp. admins via GPO or other method to resolve
service desk calls.

Comment 40

[Scoobidiver (away)]

There have been no crashes in 8.0b1 so far.

Comment 41

[Ian Gregory]

(In reply to Scoobidiver from comment #40)
> There have been no crashes in 8.0b1 so far.

I updated this morning, and Firefox still crashes. But the crash reporter is no longer catching it - I just get "Firefox has stopped working". The following is from the event log:

Faulting application name: firefox.exe, version: 8.0.0.4288, time stamp: 0x4e8342f6
Faulting module name: mozmolt0.dll, version: 7.0.1.151, time stamp: 0x4b1d8f7f
Exception code: 0xc0000005
Fault offset: 0x0000138e
Faulting process id: 0x1fd8
Faulting application start time: 0x01cc84c2267e6ad9
Faulting application path: C:\Users\GregorI\AppData\Local\Mozilla Firefox\firefox.exe
Faulting module path: C:\Program Files\Passlogix\v-GO SSO\Helper\Moz\mozmolt0.dll
Report Id: d8f78760-f0b5-11e0-9500-68b599f6eede

Comment 42

[Ian Gregory]

I ran mozregression:

Last good nightly: 2011-06-21
First bad nightly: 2011-06-22

Pushlog:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a285146675dc&tochange=b7a93f1279b7

I'm not in a position to build Firefox on this computer, however, so I can't narrow it further, unfortunately.

Comment 43

[Sorin Sbarnea]

I am working inside a corporation with over 8000 employees, and none of them is able to run Firefox. You do not get more crash reports because people do not send them or just because they do not have install privileges. 

Here is one of the reports: https://crash-stats.mozilla.com/report/index/429cbd7c-935a-4ccd-8e0e-07a222111017

Hope this will be solved soon.

Comment 44

[LegNeato]

---------------------------------[ Triage Comment ]---------------------------------

We need to get to the bottom of this and see if we're doing something wrong or Oracle is and possible mitigation steps. Marking this as tracking.

Comment 45

[Benjamin Smedberg]

In case this info didn't make it back around, Oracle is aware of this issue. It was caused by binary bits which depend on the vtable layout of nsIDOMWindow, which changed in bug 664543. It is not clear to me how they get into our address space, but it is not with a standard XPCOM component, which is why the normal version matching system doesn't catch this bug.

The registry entry in comment 33 is an effective workaround, but I don't know whether we should consider adding this during the update process or not...

Comment 47

[Scoobidiver (away)]

The mozcomp.dll crash signature is specific to Fx 7.
The nsAString_internal::ReplacePrepInternal crash signature is specific to Fx 8 and above.

Comment 48

[Ian Gregory]

I'm still getting this crash with Fx 10.0b1 - though neither the Mozilla crash reporter nor the Windows winqual system is catching it - Firefox just exits silently. I can easily work around the issue but I wonder how many users are being affected by this bug but are not visible due to the fact that crash reports have not been sent for this issue since Fx 7.0

Comment 49

[Alex Keybl [:akeybl]]

bsmedberg - Is there a public bug/issue report that we can follow to find out when Oracle plans to fix this? I also filed bug 714893 and CC'd you since I know you've looked at unreported crashes in the past.

(In reply to Ian Gregory from comment #48)
> I'm still getting this crash with Fx 10.0b1 - though neither the Mozilla
> crash reporter nor the Windows winqual system is catching it - Firefox just
> exits silently. I can easily work around the issue but I wonder how many
> users are being affected by this bug but are not visible due to the fact
> that crash reports have not been sent for this issue since Fx 7.0

Ian - did the workaround in https://bugzilla.mozilla.org/show_bug.cgi?id=680927#c33 fix your issue?

Tracking for FF10/11/12 in case this is in fact a major unreported (and unfixed) crasher and we need to take action as bsmedberg suggested:

> The registry entry in comment 33 is an effective workaround, but I don't
> know whether we should consider adding this during the update process or
> not...

Comment 50

[MikeBY]

The action bsmedberg suggests: implementing registry change documented in #33 above as part of the update process is inappropriate.

1) The registry change is a registry entry for Tivoli/Oracle ESSO, not for FF, and could have unknown consequences for that software.  Anyone care to regression test all the various flavors of Tivoli/v-GO/Passlogix/Oracle ESSO ?

2) The 'workaround' disables support for Mozilla.  When the bug gets addressed, how would you turn support back 'on' and when?  If the user's original installation of Tivoli/Oracle ESSO was custom and Mozilla support was initially off, turning it on at some later date via update would also be inappropriate.

3) A better workaround is to install SSO with custom parameters or use 'add/remove programs' to launch the Tivoli installer in Modify mode to remove Mozilla support support internally within the Tivoli installation.

4) Unreported crashes in 10b (comment #48) indicates a new issue with crash reporting.  

Keep in mind that the problem occurred with a FF update, not a change in Oracle's side.  Only bother to fix this if you want to see FF adopted in the corporate environment.

Comment 51

[Ian Gregory]

(In reply to Alex Keybl [:akeybl] from comment #49)
> Ian - did the workaround in
> https://bugzilla.mozilla.org/show_bug.cgi?id=680927#c33 fix your issue?

I haven't been able to test the registry workaround as the machine I encounter the issue on is a corporate build and I don't have permission to edit the registry - my daily routine involves closing Oracle ESSO before starting Firefox. 

However, shutting down Oracle ESSO allows Firefox to start cleanly. If I start Oracle ESSO and then start Firefox, I see the taskbar icon animate to the 'running' state then back to the 'inactive' state 2-3 seconds later.

Comment 52

[Benjamin Smedberg]

I don't think there is any action that we can take on this bug. Oracle was using an internal interface without QI-checking it, and we're at the point now where I don't think we can safely un-do that change even if we wanted to. I agree that the registry change is probably not an appropriate action to take on our part, although Oracle should perhaps recommend or ship a hotfix.

There is not a public issue report, and other than a private email saying that they were aware of the issue I don't know of an internal oracle issue ID either.

Comment 53

[Ian Gregory]

Maybe the issue can't be avoided, but is there any action that can be taken to improve the user experience? I guess resolving bug 714893 will at least offer some UI to indicate Firefox has crashed (and the Sorocco page would lead inquisitive users here), but is there any possibility of adding a message that would be shown if the Oracle software is detected at startup to indicate that there is a known incompatibility?

Comment 54

[Alex Keybl [:akeybl]]

(In reply to Ian Gregory from comment #53)
> Maybe the issue can't be avoided, but is there any action that can be taken
> to improve the user experience? I guess resolving bug 714893 will at least
> offer some UI to indicate Firefox has crashed (and the Sorocco page would
> lead inquisitive users here), but is there any possibility of adding a
> message that would be shown if the Oracle software is detected at startup to
> indicate that there is a known incompatibility?

I think this is outside the scope of FF10/11, and our best bet is to get a low-risk fix of 714893 into the build. If the crash volume is very high, we'll again get in contact with Oracle with the new numbers. Other than that, I agree with Benjamin that there's little more to do from our side.

FYI you've got an open question in https://bugzilla.mozilla.org/show_bug.cgi?id=714893#c4 which would greatly increase the chances of its resolution prior to our next release.

Comment 55

[Ian Gregory]

Thanks, I've added a further comment to bug 714893. 

I've done some further testing, and it appears in the Fx 10 builds Firefox is actually hanging indefinitely rather than crashing - on startup the process stays in the background consuming CPU. If ESSO is started whilst Fx is already running, the Fx window freezes, greys and displays the "Not Responding" window title. So I may not be able to get a crash stack.

Comment 56

[Ian Gregory]

For information, I've posted a stack trace taken at the time of the hang in bug 714893

Comment 57

[Rabs210]

Unfortunately, this continues to be an issue in FF9.0.

I agree that the workaround mentioned in comment #33 above is inappropriate. From a personal perspective, I'm now unable to launch FF. Very frustrating.

I'm utilising a corporate desktop, without Administrator privileges. Without the ability to edit the registry for SSO, FF crashes on start.

Comment 58

[Scott]

Fix from Oracle available:

FireFox 7 or 8 Crashed When ESSO-LM Agent Is Running On Client Computer 
--------------------------------------------------------------------------------

Applies to: 
Oracle Enterprise Single Sign-On Suite - Version: 11.1.1.2.0 to 11.1.1.5.0 - Release: 11g to 11g
Information in this document applies to any platform.

Symptoms
Launching Firefox 7 or 8 with ESSO-LM agent running results in Firefox crashing.

Cause
An internal modification was made to Firefox 7 and 8 that not was immediately supported by ESSO-LM.

Solution

This issue occurs with ESSO-LM versions 11.1.1.2.0 and 11.1.1.5.0.  It has been resolved in version 11.1.1.5.1 (bundle pack 1).  The preferred method of remediation is to upgrade to bundle patch 1 (11.1.1.5.1) and apply bundle patch 2 (11.1.1.5.2).

Alternate:
1 - Run the registry editor (regedit.exe) and navigate to the following key:

For a 32 bit operating system: 
HKLM\Software\Passlogix\Extensions\AccessManager\MozHO\molten\{ec8030f7-c20a-464f-9b0e-13a3a9e97384} 

For a 64 bit operating system: 
HKLM\Software\Wow6432Node\Passlogix\Extensions\AccessManager\MozHO\molten\{ec8030f7-c20a-464f-9b0e-13a3a9e97384}


2 - Right click on the key "{ec8030f7-c20a-464f-9b0e-13a3a9e97384}"

3 - Choose "new" and then choose "key".

4 - Type in the value "4.0".

5 - Right click on the new "4.0" key.

6 - Choose "new" and then choose "string value".

7 - Type in the value "Filename".

8 - Do not open it and provide any value data.  Leave it blank.

9 - Close the registry editor.

The issue should be resolved.

Comment 59

[Sheila Mooney]

Can someone who could repro this verify it's fixed?

Comment 60

[Rabs210]

Scott,

Your comments in #58 don't resolve the problem for users without Administrative privileges who are operating within a corporate environment. 

Given that the problem resulted from a upgrade / change to FF, the resolution should ideally come from there.

Comment 61

[Scott]

Oracle has implemented a fix for this.  Clearly there are more people running Firefox than are running Oracle's ESSO.  A fix from FF is unlikely IMHO. 

Regarding the Admin privileges -- if your organization has deployed ESSO, then let the IT group know that a fix is available and they should upgrade to the latest release.  No edits to the registry are needed with the latest verion.

As for testing it, I personally installed bundle patch 1 (11.1.1.5.1) and applied bundle patch 2 (11.1.1.5.2) on Windows 7 and it works fine with FF 9.0.1 (even though the SR resolution note that I copied above only lists FF 7 & 8).

Comment 62

[Rabs210]

Unfortunately I work for a large company. They have a frustrating policy of remaining two releases behind the latest version of any software release. As a result, it will be some time (months or years) before versions 11.1.1.5.1 or 11.1.1.5.2 are rolled-out.

Any upgrade to our ESSO version would affect 30,000+ users. It would also never be implemented solely to resolve a FF issue on the basis that FF is not a supported browser. Of course this is frustrating, but I'm sure this situation will be the same for many other users.

Comment 63

[Alex Keybl [:akeybl]]

(In reply to Rabs210 from comment #60)
> Given that the problem resulted from a upgrade / change to FF, the
> resolution should ideally come from there.

This is not a Firefox issue - it's an Oracle issue that they're aware of and have fixed. We can't make counterintuitive software decisions (fixing issues in 3rd party software) based upon corporate policies.

Comment 64

[Simona Badau, Desktop QA]

I'm removing the qawanted keyword since the required information in Comment 6 was provided in Comments 8 and 42. Please re-add it if anything else is still needed from the QA side.

Checked also the Socorro reports and I found several crashes that are related with the last signature of this bug on the latest Firefox versions as follows http://bit.ly/15X8gHb:
[@ nsAString_internal::ReplacePrepInternal(unsigned int, unsigned int, unsigned int, unsigned int) ]
- Firefox 24 - 14  
- Firefox 25.0 beta - 9
- Firefox 26.0a2 - 0
- Firefox 27.0a1 - 4.

Comment 65

[Ian Gregory]

I can confirm that the symptoms are no longer seen with ESSO v11.1.2.1.0

Comment 66

[Loic]

Could we close that bug report as Oracle fixed the issue on its side?