Ver. 2023.2 - SharePoint throttling and ReplaceMagic - new functionality

What is new in 2023.2:

• Fixed problem when on DFS share processing is stopping after first folder found with Access Denied permissions.
  As outside of ReplaceMagic components are used for creation of list of files for scanning, we cannot show skipped folders directly in ReplaceMagic but AccessDeniedLocations.log file will be created in c:\users\username\appdata\roaming\replacemagic\logs where you can find what was skipped. If no file is found that means that there were no locations with Access Denied. Failback location is c:\Temp.
• Improved check of relative links with documents in SharePoint.
• Extended filtering of temporary files (starting with ~$) uploaded to the SharePoint.
• Added option to remove parameters from URL which is useful in case of migrations to SharePoint when there are URLs like: https://old.system.com/sites/newsitename/path/to/file/somefile.xlsx?d=w152a583596ef4cc198abe15ad350117d as after migration parameter d will point to old document ID which will not work anymore. To set parameters for removal use Configuration => Misc:
  
  What is new?
  With this configuration ReplaceMagic will remove d and p parameters from URL. Separator between parameters is | and this will work only for links in Hyperlinks section. Replacement is not case-sensitive so if you specify d=, ReplaceMagic will replace parameter d= but also D=.
  Prerequisite: you will need to provide search&replace strings as otherwise replacement process cannot start and changes in hyperlinks need to be selected.
• Further reduction of requests to SharePoint to reduce workload triggered by ReplaceMagic.
• Fixed bug that in rare cases under heavy throttling ReplaceMagic can close.
• In SharePoint article https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online (last update from 03/20/2023; page is regularly updated) Microsoft is stating that beside user throttling there will be application throttling also. Every application will have own limits per tenant no mater if Microsoft Graph, CSOM or REST is used (ReplaceMagic is using CSOM and REST). Limits are as following (taken from MSFT site):
  
  Also different resource unit costs per request are defined:
  
  (example is for Microsoft Graph, where CSOM and REST are even more "expensive"). To translate to ReplaceMagic terminology - scanning is "easy" request as we need only to download file but in case of replacements multiple requests are required for:
  - Meta data preparation
  - Download file
  - Upload changed file
  - Optionally, set meta data back (last modification date, editor, status of file, version...)
  This will cause that replacement will require more than few request which will be taken from your quota. To reduce risk of throttling, we are strongly recommending not to use more than 2 documents in parallel (default is 4) as there will be great chance that you will be throttling. In case of throttling ReplaceMagic will automatically reduce parallelism to 1 (sequential processing) and if parameter Optimize workload is selected ReplaceMagic will start to increase parallelism after defined number of processed documents (new value 100 (reduced from 500) without throttling.
  According to Microsoft documentation, when throttling is happening there has to be used Retry-After parameter which will stop processing for SharePoint requested duration value.
  
  
• Added parameter "Use SharePoint retry-after recommendation" - when SharePoint throttles you, it will send duration during which processing cannot continue. By using this parameter, ReplaceMagic will stop processing for SharePoint requested period and continue afterwards. Default: selected.
  
  
• In case that you do not want to use SharePoint retry-after parameters values via parameter "Throttling retry-after period" you can define own wait period (default: 120). This parameter is only visible if parameter "Use SharePoint retry-after recommendation" is unchecked:
  
  Due to this parameter when ReplaceMagic is first time throttled, there will be pause of 120 seconds before processing can continue and you will see that information in progress bar. Using this and above parameter we are protecting your processing to be totally stopped and to have as less skipped documents due to throttling as possible.