Whole Tomato Software Forums
Whole Tomato Software Forums
Main Site | Profile | Register | Active Topics | Members | Search | FAQ
User name:
Password:
Save Password
Forgot your password?

 All Forums
 Visual Assist
 Technical Support
 Parsing whole solution very often
 New Topic  Reply to Topic
 Printer Friendly
Previous Page
Author Previous Topic Topic Next Topic
Page: of 2

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Jun 24 2020 :  12:31:21 PM  Show Profile  Reply with Quote
I have found a situation where VA re-parses the entire Unreal Engine directory tree for the github version of Unreal Engine, but for me this is only happening after using the github source code compiled version of the Unreal Editor to create a new game.

When I do this, instead of loading the Unreal Editor with the game project, as expected, instead I am launched directly into VS2019 with the C++ solution source code loaded. At this point VA does a full parse of the Unreal Engine tree.

However if I then close the IDE, and Unreal Editor, and either load a new instance of Visual Studio, or of Unreal Editor, and then load one of these github source editor produced game solutions, there is no extra parsing at all.

I am only working with one instance of the IDE open at a time, since VA is designed to maintain a separate symbol database for each separate instance of Visual Studio that is used, so this would require a full reparsing of Unreal Engine for each new instance and symbol database that has never before parsed Unreal Engine.

How does this compare with your experience? Obviously you are seeing something different, so how should I change my tests to try and reproduce what you are doing and seeing?

zen is the art of being at one with the two'ness
Go to Top of Page

Jorge24
Senior Member

USA
28 Posts

Posted - Jun 24 2020 :  6:38:44 PM  Show Profile  Reply with Quote
Thanks for continuing to test this. I'm having a hard time understanding the process you followed on your end. Did you say that the entire source-compiled UE must be parsed at least once for each project that references it? or that dosnt happen to you?

So, if you create 10 new UE blank project from the source compiled UE editor, will it parse the entire engine again for each project? and if so, does that only happen with the source compiled one for you or also with the Epic Launcher version?
Go to Top of Page

Jorge24
Senior Member

USA
28 Posts

Posted - Jun 24 2020 :  6:42:12 PM  Show Profile  Reply with Quote
quote:
Originally posted by feline

When you load your solution in Visual Studio, how are you doing so?

Do you first load Visual Studio, then open the Unreal solution you are working on?
Do you launch the Unreal Editor first and open the solution from here?



I launch Visual Studio and then I click on one of my pinned solutions. Then I click the green arrow to run the UE editor.





Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Jun 25 2020 :  05:19:37 AM  Show Profile  Reply with Quote
I am basically doing the same thing here... so why the difference?

If you just load your solution, as shown in the screen shot above, and then wait a few moments before running the UE editor, does VA start doing a full parse of Unreal Engine? Or does VA notice that nothing has changed, so there is no need to do a reparse?

I am trying to work out if something seems to be triggering the extra parsing.

Do you work with more than one instance of Visual Studio open at once, or just the one instance?

zen is the art of being at one with the two'ness
Go to Top of Page

Spanky
New Member

Canada
2 Posts

Posted - Aug 30 2020 :  09:14:47 AM  Show Profile  Reply with Quote
I was also having this same problem and I can confirm that creating the junction does fix the issue for me.

I also do not have the launcher/engine installed on my C drive. I have it installed in "D:\Code\Epic Games".

I created a new project here "D:\Code\Unreal Projects\MPTesting". When I open this, VAX will end up parsing all of Unreal again which takes a while (I have created multiple projects at this point). Also checking my 'System include directories' in VAX didn't list the Unreal directories.

I created the junction and opened the vcxproj and vcxproj.filters files (on my D drive) and updataed the directories in both of those to point to the path through the C drive junction. I cleared the VAX database and reopened the project in Visual Studio. Unfortunately this does not work as the Engine project (the Unreal engine code) within my project solution is still referencing the files on my D drive. I did however notice that the 'System include directories' now does list all of the Unreal directories properly.

I then launched the UE4Editor.exe from the junctioned path (C:\Program Files\Epic Games\UE_4.25_source\Engine\Binaries\Win64\UE4Editor.exe) instead of from my D drive (D:\Code\Unreal Engine\UE_4.25\Engine\Binaries\Win64\UE4Editor.exe). I told Unreal to open the same project (D:\Code\Unreal Projects\MPTesting) and this time it informed me that the project was generated with a different version of the engine. I guess it matches the path somehow. I ended up converting the project in place (people should create a backup somehow or do the conversion to copy option here as this process may fail and render your project broken).

After a bit, the project opened in Unreal properly. I then clicked on File->Open Visual Studio from within Unreal. It then began parsing all the system includes and I could see it scanning the Unreal files on the C drive now. The Engine project is also referncing files on the C drive rather than the D drive. The Unreal symbols are now in italics

I then created another project (still launching Unreal from the junction on C) and opened it in Visual Studio. After a noticably longer 'Loading system symbols' process, it quickly parsed my project files (1 or 2 seconds) and all highlighting was then working properly with Unreal symbols now in italics. It did not parse any of the Unreal files again. As long as I launch everything from the junction path rather than where the files are actually stored, things work properly and it does not reparse Unreal every time. I would have to convert over any existing projects to use the new path which may or may not be viable for some people.
Go to Top of Page

Spanky
New Member

Canada
2 Posts

Posted - Aug 30 2020 :  09:18:12 AM  Show Profile  Reply with Quote
And to be clear, both the Unreal install and my projects still exist on my D drive. I can open my projects from their D drive path (once they are converted obviously). I only need to launch the Unreal Editor through the junction path (C drive) and not it's actual path (D drive).
Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Aug 31 2020 :  08:54:18 AM  Show Profile  Reply with Quote
Thank you for the update, I am glad that the junction is helping. This is a work around for now.

The basic problem is that Unreal Engine is complex / hard for VA to handle correctly. You can easily have several different versions of the Engine installed at the same time, in different locations. Since the engine is listed as part of the solution you are opening, its really not obvious that this is actually a stable external library and should be treated differently to the rest of your solution.

The more I look into this problem, the more strange edge cases I seem to encounter.

zen is the art of being at one with the two'ness
Go to Top of Page

scottarius
Starting Member

USA
1 Posts

Posted - Nov 16 2020 :  2:08:07 PM  Show Profile  Reply with Quote
Is there any updates on this? I just recently bought VA for Unreal Dev and the constant re-parsing is killing me.

I'm using a source build of 4.25. All the ue4 features seem to work fine otherwise. Everything is on my C drive, so I don't think this junction fix will apply. The .uproject certainly is using the correct path for my compiled editor. I normally open the .sln file directly, not via the UE4 Editor. Sometimes it will re-parse everything, sometimes not. I can't for the life of me figure out what triggers it.

I do have other versions of UE4 installed for different projects, and they all seem to have this reparsing issue, but everything is pretty separated.
Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Nov 17 2020 :  08:57:16 AM  Show Profile  Reply with Quote
With one of your Unreal Engine solutions open, can you please look at:

VA Options -> C/C++ Directories

and check the list of stable include directories. Does this include all of your Unreal Engine sub-directories? You should should see every single sub-directory under the base directory for Unreal Engine in this list.

Even if the source code was downloaded to C drive, you can still run into the problem where VA does not realise that this is the unreal engine source code, depending on which directory the source code is based under. So this test should tell us if VA thinks that this is a stable set of directories for Unreal Engine or not.

zen is the art of being at one with the two'ness
Go to Top of Page

amoldeshpande
Senior Member

USA
33 Posts

Posted - Nov 26 2020 :  09:45:24 AM  Show Profile  Reply with Quote
quote:
Originally posted by feline

Thank you for the update, I am glad that the junction is helping. This is a work around for now.

The basic problem is that Unreal Engine is complex / hard for VA to handle correctly. You can easily have several different versions of the Engine installed at the same time, in different locations. Since the engine is listed as part of the solution you are opening, its really not obvious that this is actually a stable external library and should be treated differently to the rest of your solution.

The more I look into this problem, the more strange edge cases I seem to encounter.



Not trying to be a jerk here, but have you considered looking at HKLM\Software\EpicGames\UnrealEngine\4.xx for the install location and then parse the project file to find the version being used in the project to match ?

The toolchain architecture that Epic makes tends to be stable for years of an Engine version's life and major changes would roll out very slowly. Enough time to revisit if needed.
Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Nov 26 2020 :  11:22:53 AM  Show Profile  Reply with Quote
Unfortunately it's not that simple. I have one machine that has 2 or 3 different versions of Unreal Engine installed via the Epic Games launcher, it also has a separate version of the Unreal Engine source code downloaded from Github and compiled.

Checking on this machine, the registry key lists 5 different install locations, for 5 different versions of Unreal Engine, but there is no mention at all of the version built from the downloaded source code.

zen is the art of being at one with the two'ness
Go to Top of Page

amoldeshpande
Senior Member

USA
33 Posts

Posted - Nov 26 2020 :  12:21:22 PM  Show Profile  Reply with Quote
quote:
Checking on this machine, the registry key lists 5 different install locations, for 5 different versions of Unreal Engine, but there is no mention at all of the version built from the downloaded source code.


Well, obviously that's under HKCU\Software\Epic Games\Unreal Engine\Builds\someguid
Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Nov 26 2020 :  12:28:35 PM  Show Profile  Reply with Quote
Ah, I knew it had to be somewhere

Defining the location of the Unreal Engine tree sounds really simple, and in simple cases it is. Unfortunately once you get into complex cases, I have yet to pin down a good, simple rule that works well. Some of our users are also working with customised versions of Unreal Engine, so simply relying on features of the "original" Unreal Engine tree breaks down in these cases.

If you have a good plan I would certainly be interested!

zen is the art of being at one with the two'ness
Go to Top of Page

amoldeshpande
Senior Member

USA
33 Posts

Posted - Nov 26 2020 :  1:08:03 PM  Show Profile  Reply with Quote
Well, I don't know the ins and outs of VaX, but every UE4.vcxproj seems to have a BuiltWithUnrealBuildTool configuration (or you could just go by project file name) to detect that this is a UE4 project. Within the project file, there seem to be OutDir and IntDir properties which point to the engine version in use by that project.

If all we're interested in is finding the UE4 source tree for a given project.

Rooting around in project files may seem weird, but it's likely to be a pretty stable approach since Epic doesn't usually go nuts changing things in the toolchain. You could probably ask on their licensee forums (or one of the users here who has a license could) if they have another, recommended way to detect the source tree.



Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Dec 02 2020 :  12:19:32 PM  Show Profile  Reply with Quote
This sounds interesting. Unfortunately I have just opened 5 different copies of "UE4.vcxproj" from 5 different Unreal Engine game projects I have on one of my test machines.

Only 3 of these files have the BuiltWithUnrealBuildTool section. Where the section is present it does seem really rather helpful.

However the other 2 files have no such section. These are two different game projects created from the git source code downloaded and compiled version of Unreal Engine. I have even done a find in files across the project directory tree for the string "BuiltWithUnrealBuildTool", just to be sure, and no references.

In part the problem stems from the fact that Unreal Engine breaks one of our basic assumptions about solution structure, which is that your solution doesn't actually contain stable include directories as one of the projects. Normally this is a reasonable assumption, but not here.

zen is the art of being at one with the two'ness
Go to Top of Page

grahfoster
New Member

USA
2 Posts

Posted - Dec 09 2020 :  2:30:27 PM  Show Profile  Reply with Quote
Just wanted to chime in that I'm having this problem as well. I'm on the trial version of VA and plan to purchase the full version when that runs out, regardless of whether this problem is fixed, but it would be great to get it ironed out eventually if possible.

Here is my system info copied from the VA options menu:
License: trial
VA_X.dll file version 10.9.2393.0 built 2020.10.28
DevEnv.exe version 16.8.30717.126 Community
msenv.dll version 16.0.30717.112
Comctl32.dll version 6.10.18362.1198
Windows 10 10.0 1909 Build 18363.1198
8 processors (x86-64, WOW64)
Language info: 1252, 0x409

Platform: Project defined
Stable Includes:
// whole lotta stuff

I have verified that I have Unreal Engine support enabled and that my stable includes are all there. Generally what happens is that every two days, or even within 24 hours, VA will have to re-parse all the files in my solution, which usually takes around 15 to 20 minutes. The last time this happened, I had literally done nothing to the project in the intervening time (though I'm sure some automatic updates could have happened to affect something somewhere). I'm currently running Unreal 4.26 but this problem was present on 4.25 as well. Let me know if there's any more info I can provide to help you find a fix for this. I haven't had any recent Visual Studio or Unreal crashes, so I know it isn't related to that. Just curious what could be provoking the re-parse.

Forgot to mention that during the reparse, VA can still recognize Unreal's symbols. It's just the autocomplete and a few other features that I end up having to wait around for.


Edited by - grahfoster on Dec 09 2020 2:32:20 PM
Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Dec 10 2020 :  06:41:22 AM  Show Profile  Reply with Quote
Do you see "all" of your Unreal Engine sub-directories in the stable include directories list? Checking they are all there is a pain, so checking if you have a long list of them is ample.

What do you have the following set to:

VA Options -> Projects and Files -> Upon opening a file, parse all other unparsed files in the same directory:
if Solution is empty
if Solution is not empty

VA Options -> Performance -> Parse all files when opening a project

Are you seeing the status bar say "VA: parsing xxx"? I am just double checking that VA is actively parsing everything, and its not the IDE intellisense parser running instead.

This should not be happening, but clearly something is going on here.

zen is the art of being at one with the two'ness
Go to Top of Page

grahfoster
New Member

USA
2 Posts

Posted - Dec 10 2020 :  4:14:59 PM  Show Profile  Reply with Quote
Thanks for responding! I currently have "if Solution is empty" checked. I also have "Parse all files when opening a project" checked, which seems to be the default option. I didn't know what behavior to expect if I unchecked this (specifically, I worried about new files not being parsed). Also, if that box is unchecked, what happens when I generate a new solution for a new Unreal project?

It does definitely say "VA: Parsing." I have disabled Intellisense and even encountered this problem while the database was disabled, though I later restored it for convenience when it didn't improve anything.

As a side note, VA loaded my solution just fine today in about 30 seconds without having to parse.
Go to Top of Page

amoldeshpande
Senior Member

USA
33 Posts

Posted - Dec 10 2020 :  4:21:03 PM  Show Profile  Reply with Quote
quote:
Originally posted by feline

This sounds interesting. Unfortunately I have just opened 5 different copies of "UE4.vcxproj" from 5 different Unreal Engine game projects I have on one of my test machines.

Only 3 of these files have the BuiltWithUnrealBuildTool section. Where the section is present it does seem really rather helpful.

However the other 2 files have no such section. These are two different game projects created from the git source code downloaded and compiled version of Unreal Engine. I have even done a find in files across the project directory tree for the string "BuiltWithUnrealBuildTool", just to be sure, and no references.

In part the problem stems from the fact that Unreal Engine breaks one of our basic assumptions about solution structure, which is that your solution doesn't actually contain stable include directories as one of the projects. Normally this is a reasonable assumption, but not here.



There is nothing in the two project files that points to which version of Unreal Engine they are using ? That seems strange. What do their OutDir and Intdir properties look like ?


In any case, can you please at least optimize the more general case where we are not building UE4 from source ? Every time I create a new UE4 project VA wastes 30 minutes parsing the already parsed version of UE that I am using.

Even just matching the path prefix from the registry entries should be sufficient for the general case.

I'm getting really tired of my computer being unusable because VA doesn't respect the multiprocessing limits. If the UE4 indexing was fixed at least that annoyance would be smaller.
Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Dec 11 2020 :  11:26:24 AM  Show Profile  Reply with Quote
amoldeshpande, better support for Unreal Engine location is a high priority bug report, and it is something we would like to get working fairly soon.

There are definitely clues to the location of unreal engine in the project files, the problem is that there are various special cases we have to deal with when working with unreal engine. It seems simple at first, but the more you look, the more odd situations you tend to encounter.

For now, if you are able to use the file system junction point then this seems to be a solid work around, at least in all of the tests I have run so far. If you are having problems with this I am more than happy to look at this with you.

For VA parsing impact, have you tried any of the settings on VA's Performance page? Specifically:

VA Options -> Performance -> Enable multithreaded parsing
VA Options -> Performance -> Limit the maximum number of concurrent threads to X (requires restart)
VA Options -> Performance -> Thread priority = Normal / Below Normal / Lowest (requires restart)

I have changed the number of concurrent threads to 3 out of 8 (8 core system) and triggered a VA symbol database rebuild, and loaded an Unreal Engine solution, for VA to parse.

Using the build in Windows 10 Resource Monitor reports that devenv.exe is now using 38% of the CPU time on my system, in line with this setting, so these settings should be working for you, to limit the CPU impact of VA parsing everything.

zen is the art of being at one with the two'ness
Go to Top of Page

amoldeshpande
Senior Member

USA
33 Posts

Posted - Dec 11 2020 :  12:12:26 PM  Show Profile  Reply with Quote
I don't understand what's hard about "Have I indexed a file in a location beginning with c:\something\unrealengine\4.25" before ? It should have nothing to do with parsing project files or trying to figure out install locations.

It doesn't have to be your ideal or final solution to do it this way. Just an interim solution.

It is incredibly frustrating that you guys seem to be waiting (for almost 2 years now) to figure out the most fancy ideal solution, which will probably never materialize.

As far as CPU goes, I don't want to hijack this thread, I'll open a different one.


Edit: I don't view junction points as an acceptable workaround. Too much Windows software gets confused with links and I don't want to deal with that headache.

Edited by - amoldeshpande on Dec 11 2020 12:13:24 PM
Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Dec 12 2020 :  07:52:57 AM  Show Profile  Reply with Quote
Apologies, I have not explained the issues here very clearly. There are two separate effects at work here, but they overlap.

By default, VA does not consider directory trees that are part of your solution to be stable include directories. So if you open two different solutions that both point at Unreal Engine, then both solutions will want to parse everything from scratch, or at least check all of the files and see if they need to be parsed. This is based on the, generally reasonable, idea that different solutions don't have that much overlap.

Unreal Engine, being both a stable external library, and part of the solution at the same time, breaks these assumptions.

VA should not parse everything in unreal engine all of the time, regardless, if you are just opening the same solution again. VA will check to see if the files are different than last time, but that should be all that happens. So if we have excess parsing then this is a bug, regardless.

The junction points are used to make sure that VA's "you are working with unreal engine" checks work correctly, which enables certain unreal engine specific features. This also helps with parsing speed, but only when moving between different solutions, since we know that the entire unreal engine project, and all of its files, don't need to be checked when parsing the solution, since they are a stable external library.

I hope this makes a bit more sense.

zen is the art of being at one with the two'ness
Go to Top of Page

feline
Whole Tomato Software

United Kingdom
18724 Posts

Posted - Dec 12 2020 :  08:00:15 AM  Show Profile  Reply with Quote
grahfoster, back to your questions.

The "if Solution is empty" settings are for situations when you are just using the IDE as an editor, or are using an external compiler, so there is no solution to use to get a list of all of the files and directories you are working with. In this situation you can tell VA to parse other files in the same directory, so VA will pick up as you open / reference files.

If you are working with a solution, then VA monitors the solution, and projects, and detects when new files are added, and parses them as it goes along.

A new unreal project generates a solution, so all of the files we should parse are known when the IDE loads the solution, so everything should "just work".

The status bar message proves VA is parsing, as expected. But I don't understand why this only happens some of the time.

Have you used the setting:

VA Options -> Performance -> Use alternative directory for symbol database (requires restart)

to change the location of your symbol database? This should not matter at all, but I am just wondering if some other process could be interacting with VA's symbol database, for example anti-virus software?

You might want to try turning Off the setting:

VA Options -> Performance -> Parse all files when opening a project

to see if this helps much. This isn't normally used, but is useful when you have a very large solution, and don't want VA to check for changes everywhere, and just focus on the files you are editing and directly including.

zen is the art of being at one with the two'ness
Go to Top of Page
Page: of 2 Previous Topic Topic Next Topic  
Previous Page
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
© 2023 Whole Tomato Software, LLC Go To Top Of Page
Snitz Forums 2000