| T O P I C R E V I E W |
| jxtan |
Posted - Jun 30 2008 : 1:49:26 PM
Our project is huge, over few thousand cpp/c files and few thousand hpp/h files, well over million lines of code.
When project is opened, visual assist would take ages to process. I also noticed there's huge IO processing take in place as well.
After project is opened and visual assist completed the parsing, I/O read on devenv.exe is 22GB, I/O write is 21.8GB, I/O write was at excessive of 400 times/second. I don't think our source code is at 22GB C++ files, maybe total no more than 200MB of source files.
As result, the entire harddrive was taken over by visual assist, and nothing else can run, I can't open and run any other programs.
Suggestion:
- reduce overall numbers of I/O read/write by cache them manually in memory, because HDD has good throughput but doesn't respond well to large number of read. Windows' cache works well in most cases, but doesn't seem work well in this.
- create a pause option to pause the parsing if needed.
|
| 16 L A T E S T R E P L I E S (Newest First) |
| jxtan |
Posted - Jul 03 2008 : 2:57:51 PM
Log file contain partial source code. I need to get someone to approve this first. May take few days.
|
| sean |
Posted - Jul 02 2008 : 9:35:04 PM
Do you run multiple instances of the IDE concurrently?
If not, logging might give us some clues. You can create logs per the instructions here:
http://docs.wholetomato.com?W305
Enable logging before you open the solution.
thanks
|
| jxtan |
Posted - Jul 02 2008 : 2:10:32 PM
feline:
Yes, I used the process explorer to find out the I/O data. How big is all of your source code for 330MB read, 240MB write.
I used the Proces Monitor to trace the devenv.exe, and found following
- a lot small write to files such as
C:\\Documents and Settings\\%USER%\\Local Settings\\Application Data\\VisualAssist\\vs8\\Proj_%NUMBER%\\Db3Idx.db
C:\\Documents and Settings\\%USER%\\Local Settings\\Application Data\\VisualAssist\\vs8\\Proj_%NUMBER%\\Db3Ds.db
C:\\Documents and Settings\\%USER%\\Local Settings\\Application Data\\VisualAssist\\vs8\\Proj_%NUMBER%\\VA_S.tmp
each write is about length around 20byte to 140byte, mostly about 20-40byte.
- a lot small write to the same location in those files under C:\\Documents and Settings\\%USER%\\Local Settings\\Application Data\\VisualAssist\\vs8\\Proj_%NUMBER%\
- writting to C:\\Documents and Settings\\%USER%\\Local Settings\\Application Data\\VisualAssist\\vs8\\Proj_%NUMBER%\\Db2Idx.db is not cached, and paged IO, could be very slow. Is this file memory mapped?
- majority of the read and write are under C:\\Documents and Settings\\%USER%\\Local Settings\\Application Data\\VisualAssist\\vs8\\Proj_%NUMBER%\
- read of the source file is very small part of overall read.
|
| gmit |
Posted - Jul 01 2008 : 3:00:02 PM
jxtan, maybe you could use filemon utility from sysinternals with filter set to vs2005 process name - using it, you could see which file is accessed the most during project opening.
|
| feline |
Posted - Jul 01 2008 : 2:38:14 PM
How are you measuring the I/O ?
I have just run some tests under win2k with VS2005 and VA 1640. I am using Process Explorer:
http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
and turn on the "I/O Reads", "I/O Read Bytes" and "I/O Write Bytes" columns.
I have renamed feacp.dll so this is not a factor. I told VA to rebuild its symbol database and then restarted the IDE and loaded my large test solution. This is made up from the Firefox source code. It holds 8,891 files according to VA's Open File dialog, and takes around 140meg of hard drive space for the source code.
So this should be a reasonable reference point for your experience.
It took around 15 minutes for VA to reparse all of the files, running in a Virtual machine while I was doing other heavy duty work on the machine.
My I/O figures from Process Explorer are:
I/O Reads = 194,527
I/O Read Bytes = 329,737,438
I/O Write Bytes = 234,763,144
Now unless I have made a silly mistake that gives me about 330meg of read and 240meg of write, which is reasonable considering the amount of source code, and the fact that I have
VA Options -> Performance -> Parse all files when opening a project
turned on.
Have I done something obviously and massively different to you? There is a massive difference between 300meg and 22gig of I/O!
My numbers should be a worst case scenario for loading the IDE since I told VA to rebuild its entire symbol database, not just reparse the solution.
Can you try excluding VA's symbol database from your anti-virus scanner and see if this makes any difference? |
| jxtan |
Posted - Jun 30 2008 : 5:54:04 PM
Not important here, but I am pretty sure rename of feacp.dll didn't do anything at all. Because our project would never load with VS intellisense. And only way for our project to load is put project.ncb as directory or readonly for both solution dir and %TEMP% dir.
Unless this has changed with VS2005 SP1 or later updates. |
| rhummer |
Posted - Jun 30 2008 : 5:49:19 PM
quote:
Renaming feacp.dll doesn't do anything in VS2005, the intellisense is done from different place now.
That seems incorrect. I renamed that DLL and I never see Visual Studio updating the Intelliesence message or progress bar, and I use VS2005. |
| jxtan |
Posted - Jun 30 2008 : 5:43:31 PM
Regenerated the solution, opening it again (no source changes), I am seeing the devenv.exe do I/O Delta Writes of 2400->6000 times/second |
| jxtan |
Posted - Jun 30 2008 : 5:40:29 PM
> What is this second number?
8007
The %TEMP% also have the same thing put in, where I put a directory name same as project.ncb.
Renaming feacp.dll doesn't do anything in VS2005, the intellisense is done from different place now.
Visual Assist only seem writes to C:\\Documents and Settings\\%USER%\\Local Settings\\Application Data\\VisualAssist\\vs8\\Proj_%SOME_NUMBER% directory, but that directory only is 98MB. How is it possible that it write out 21.8GB of data to disk.
------------
In addition, our work flow maybe different, our solution and projects are generated from our build system. Project/solution are *not* checked into source control. Does Visual Assist rely on the GUID of the project/solution so it doesn't reparse the entire project when opening again?
|
| feline |
Posted - Jun 30 2008 : 5:06:24 PM
jxtan in VS2005 the IDE will make a second NCB file in the %TEMP% directory if the main NCB file is read only. Does renaming the feacp.dll make any difference?
How many files do you have in your solution?
If you open VA's Open File in Solution dialog (Alt-Shift-O) the title bar contains two numbers. The first number is the number of files currently listed, which changes as you filter the list. The second number is the total number of files in the list, which is normally the number of files in your solution. What is this second number? |
| jxtan |
Posted - Jun 30 2008 : 4:36:52 PM
VA Options -> Performance -> Parse all files when opening a project
then, I don't get all the symbols which I rely on to jump from one code to another code. |
| jxtan |
Posted - Jun 30 2008 : 4:35:45 PM
Each project name is unique.
Intellisense is disabled by putting a directory name same as project.ncb, or change to readonly.
|
| feline |
Posted - Jun 30 2008 : 3:48:57 PM
Step one, can you try disabling the IDE's intellisense scanner as described here and see if this helps:
http://docs.wholetomato.com?W133
You might also find that turning Off:
VA Options -> Performance -> Parse all files when opening a project
helps. |
| hotzenplotz |
Posted - Jun 30 2008 : 3:40:51 PM
@jxtan:
Do you rename workspaces often? E.g.: at my company, most of the time I have 3-4 folders on my C drive which contain some projects. When compiling, all of them should be called "Projects", so I rename them when I switch to work on another project (or another branch). Those folders are quite large, and also contain some big libraries like Boost, Xerces etc. When I open a solution in VS after renaming my "Projects" folder, VAX will also start to parse stuff like crazy.
When I re-open a solution, also after rebooting, but without renaming the working copy, VAX doesn't re-parse my files (only if they have changed of course).
|
| jxtan |
Posted - Jun 30 2008 : 3:12:08 PM
Visual Studio 2005, Visual Assist X version 10.4.1640.0 built 2008.05.22
|
| rhummer |
Posted - Jun 30 2008 : 2:41:23 PM
What version of VA X, and Visual Studio are you using? I work with a code base that is pretty similar to yours in size and I have no problem on start up. |
