Ver. 2020.10.1 - help with SharePoint throttling and much more....

What is new in 2020.10.1:

  • Fixed issue that if SharePoint App-only authentication mode is selected client secret (password) is not in all places properly used. This authentication mode can be used in case that legacy authentication mode is disabled for your site(s). More details can be found in ReplaceMagic help article "Granting access using SharePoint App-Only (use when Legacy Authentication mode is not allowed)"
  • Updated SharePoint section of help file (SharePoint errors, App-Only authentication, MFA,...)
  • Fixed problem that PowerPoint files are skipped if "Check broken links" is enabled and there are OLE links without any entry in Address field.
  • Per default ReplaceMagic works with parallelism of 5 documents at the same time (parameter "Parallel document process" in Configuration). As newer computers usually have more than 5 threads available, meaning they can process more than 5 documents in parallel, if this number is not increased ReplaceMagic will not optimally use hardware capacity. To go around this we are introducing parameter "Optimize workload ... after every ... documents" which will after defined number of documents try to increase workload to maximum number of available threads:
  • In case of that documents are stored on SharePoint might happen that SharePoint starts to throttle processing. To reduce risk that processing is full stopped by SharePoint (too much workload) you can set ReplaceMagic to reduce parallel document processing when SharePoint starts to throttle accesses to documents. TO set that use parameter "Decrease workload during SharePoint throttling?". In case that ReplaceMagic receives error messages 429 - too many requests it will reduce number of parallel documents for half as long as those error messages are shown. Minimum reduction will be to 1 document at the same time (sequential processing). In case that you want that ReplaceMagic later increase number of parallel documents for processing please select parameter "Optimize workload..."

    Remark: if you have computer with more than 8 threads we do not recommend to use this option without parameter SharePoint safety belt  
  • New parameter "SharePoint workload safety belt" - in case that ReplaceMagic should optimize workload to use maximum what current hardware can do with this parameter we are controlling that when processing is done against SharePoint we do not use more than 8 documents in parallel to reduce risk that ReplaceMagic is throttled by SharePoint. In case your hardware has more than 8 threads and you are using "Optimize workload..." we are recommending to using this parameter:

  • Fixed problem with mm/dd/yyyy format and license file
  • In case that during replacement process with documents stored in SharePoint some of them are skipped due to following error codes:

    Document bigger than set limit
    Document processing manually stopped
    Document processing timeout reached
    Error code 429 => SharePoint Throttling
    Error code 503 => SharePoint Throttling

    you will have possibility to prepare them for reprocessing immediately after initial replacement process is done by simple click on one button.
    Steps:
    1. When replacement process is done in dashboard you will see how many documents are skipped:

    2. If you click on Additional Info tab


    You will see all documents that are skipped with reasons why.
    In case that you see documents with above mentioned errors by clicking on "Reprocess SharePoint files" ReplaceMagic will create collection of only those documents which you have to process immediately by going to Search & Replace tab where you will run everything exactly the same.

    Remarks:
    1. In case that documents where skipped due to document size they will be skipped again except if you do not change maximum size of file. See Configuration => parameter "Skip files bigger than"
    2. If documents were skipped due to timeout to be able to process them you will need to increase timeout parameter. See Configuration => parameters "Maximum time to process document" and "Wait time to close document processing".
    3. If documents were skipped due to SharePoint throttling (errors 429 and/or 503) there are two options:
    a). reduce number of documents to be processed in parallel => see Configuration => parameter "Parallel document processing". As a smaller helper you can also use "Decrease workload during SharePoint Throttling"
    b) Wait with processing few hours to allow that SharePoint lifts throttling limits. Please do not close ReplaceMagic as skipped files will be also gone.

    In case of option 1 we would recommend to set parameter "Parallel document processing" to 1 so that documents are sequentially process one by one. In case that SharePoint allows that you enable parameter "Optimize workload" as that will increase number of documents for parallel process for one every X documents. Please keep in mind that parameter "SharePoint workload safety belt" is also enabled.

    "Reprocessing SharePoint files" is visible only when selected scan location starts with http(s).

    Additional remark - you can repeat "Reprocess SharePoint files" after each replacement process, basically, as long as you do not have any document skipped with above mentioned error messages.

 

Ver. 2020.9.1 - improvements in processing Excel files and more...

Focus of this updates are improvements related to processing of Excel files like:

  • In Excel documents under Link Sources there might be link sources that are visible over Data => Connection and Link Sources which are hidden. Previously, ReplaceMagic was showing all Link Sources which is now changed that ReplaceMagic shows only visible Link Sources. In case that you want to see all Link Sources go to Configuration => Processing and select parameter "Show all Excel Link Sources". Only shown Link Sources will be used later for replacements
  • Extended Link Source result list to include indicator if link source (Excel: Data => Edit Links) is used somewhere in Excel file or not
  •  Improved usage of regular expressions when search string starts with reserved character
  • Further improvements...

Ver. 2020.8.1 - trial limits relaxed, extension for command line mode and exports...

What is new.

  • Removed trial limitation related to position of links in documents
  • Extended export after scanning documents to include also Additional Info results (documents that were skipped)
  • Extended Console entries to include counter of skipped documents plus smaller adjustments in console entries
  • Extended CommandLine Config file to use settings for skipping files, folders or files and folders based on entered strings
    (section:
    <EXCLUDEFILESFOLDERS></EXCLUDEFILESFOLDERS>
    <EXCLUDEFILESFOLDERSSTRINGS><EXCLUDEFILESFOLDERSSTRING></EXCLUDEFILESFOLDERSSTRING></EXCLUDEFILESFOLDERSSTRINGS>
    see more here)
 

 

Ver. 2020.7.2 - minor release....

What is new in ReplaceMagic ver. 2020.7.2:

  • Fixed bug - flag ReadOnly was not set back to documents that were not changed during replacement process. This is happening in case when documents are not stored on SharePoint. 

Ver. 2020.7.1 - SharePoint Keep modified date and editor and more....

What is new in ReplaceMagic ver. 2020.7.1:

  • Improved "Keep "Last modified date" and "Modified by" values" for documents on SharePoint (ReplaceMagic => Configuration => SharePoint)
  • Fixed bug that export of processing results during silent (command prompt) mode did not work
  • Save of replacement configuration - Fixed problem that settings for replacement for SharePoint were not saved
  • Extended import of previously scanned documents (SharePoint) as in case that parameter "Keep "Last modified date" and "Modified by" values" was selected "Modified by" was not kept. To enable this new export done with version 2020.7.1 will be needed.
 

 

Ver. 2020.6.2 - Improved processing of SharePoint .aspx pages and more

New in version 2020.6.2 (19th of June, 2020)

  • Fixed bug that "Maintenance expired" popup appeared with valid license and English (Canada) date format (YYYY-MM-DD)
  • ReplaceMagic is now using FIPS compliant encryption algorithms
  • Extended selection in which hyperlink area change should be done. Initially, you could say that change should be make only in Address area of hyperlink and now you can set Address or TextToDisplay:

    Default is both (nothing is selected).
  • As each hyperlink in Office documents has two properties: Address and TextToDisplay from this version changes in each of areas will be separately counted. For example, there are 2 hyperlinks:

    After ReplaceMagic makes changes of google to replacemagic you will see both changes in results grid:

    and in replacement dashboard one counter for Hyperlink.Address and one for Hyperlink.TextToDisplay:








  • Changes of SharePoint .aspx pages do not work anymore - Recently we noticed that SharePoint is blocking changes of .aspx page which worked before. After investigation it looks like that on site level Microsoft changed authorizations by including DenyAddAndCustomizePages which now blocking changes of .aspx pages (when you try to do that ReplaceMagic will show in AdditionalInfo tab under skipped documents that you did not have permissions to make changes). More details and workaround can be found in this article (PowerShell scripts).
  • Improved processing of SharePoint .ASPX, Wiki and Canvas pages in case that you have special characters or use languages like German French... To use new functionality you will need to set parameter HTML Decode/Encode SharePoint pages [BETA] under Configuration => SharePoint

Ver. 2020.6.1 - OneDrive for Business extension and more

What is new:

  • Further improvements when using Multi-Factor authentication on SharePoint (still on beta)
  • From this version we included additional parameter "Stop replacements if backup does not work" (default: selected)



    which is used if you select that ReplaceMagic has to make backup of your documents (no matter if this is backup of all or only changed documents). Basically, with this option if ReplaceMagic cannot create backup (for example, not enough disk space) documents will not be saved and even if ReplaceMagic will report that changes are done; documents will appear in Additional Info tab with message that due to the backup problem save did not work.

    In case that you do not use backup functionality this parameter will not have any impact.
  • Extended support for OneDrive for Business [BETA] (URLs like: https://replacemagic-my.sharepoint.com/personal/<upn>)
  • Fix rare bug that ReplaceMagic was crashing during maxID search when document libraries had more than few hundred thousand of documents
  • Further improvements to find maximum document ID on SharePoint
  • Reduced default value of SharePoint Query Row Limit parameter to 4900 to avoid hitting normal SharePoint threshold limit of 5000
  • In case that you set parameter "Set ReplaceMagic comment"


    ReplaceMagic will during check-in/check-out or change of moderation status use value from text field (here: "ReplaceMagic Processing"). In case that this parameter is not set ReplaceMagic will retain current document value. Previously, ReplaceMagic was putting empty string.
  • In case that you are attaching template to your Word documents ReplaceMagic can show those links and make their replacements. Attached Templates will appear under Link Sources tab (in Name column you will see Attached Template text) and if you want to make replacement in that area you will need to select replacements in Attached Template (Word):
 

 

Ver. 2020.5.4 - further SharePoint extensions

  • Further improvements when processing SharePoint locations with more than 5000 items
  • Fixed bug that PowerQueries sections was ignored when Excel documents were in SharePoint.

This upgrade is recommended if you have document locations with more documents than SharePoint threshold value (default: 5000)

Ver. 2020.5.3 - additional SharePoint support

New in version 2020.5.3 (13th of May, 2020) - ...

  • Extended "Add SharePoint site" with option to set LegacyAuthenticationMode directly.
  • Extended "Add SharePoint site" with option to jump to SharePoint configuration directly.
  • Changed coding to identify maximum document ID (relevant for SharePoint) to avoid threashold limits from SharePoint
    This upgrade is recommended if you have document locations with more documents than SharePoint threashold value (default: 5000)

Ver. 2020.5.2 - a lot of new features in ReplaceMagic

We hope that you are all safe and healthy managing to balance your private and business life in those difficult COVID-19 times.

In last 2 months we spent most of the time helping customers overcome current situation but ReplaceMagic team also actively worked to prepare new version will of new and improved features.

Version 2020.5.2 has only correction not to show error message when ReplaceMagic configuration is saved (compared to 2020.5.1).

Below you can find what is new:

    • Trial of 30 days is only for replacement process. Scanning of documents and check for broken links will not have any time limitation (or any other limitation).
    • From next release ReplaceMagic version numbering will change to format YEAR.MONTH.RELEASE -> something like 2020.5.1 which is first release in May. In case that there is new version in May next number will be 2020.5.2 or if next version is in June number will be 2020.6.1
    • Scan documents result table extended with DocIdUrl link (SharePoint only).
    • PowerQueries in Excel - ReplaceMagic supports changes of source in Excel PowerQueries:

 

  • During generation of XML configuration for command prompt run all XML reserved characters are encoded to enable proper XML formatting. During start of ReplaceMagic with XML configuration file encoded characters will be decoded back to what you initially provided.
  • Extended logging of errors during test of SharePoint connections. Options "Create log file" and "SharePoint developer log info (enable only on ReplaceMagic team request)" from Configuration => Log have to be enabled.
  • During preparation for scanning of SharePoint location it is required to find maxID of documents in selected location. To accelerate identification process we included parameter MaxID range factor:

    Default value is 2 meaning that upper limit everything when there are no documents in given range will increase according to the formula => NewUpperLimit = UpperLimit + 2*UpperLimit. As this sometimes might not be enough or is too much you can adjust this parameter. By setting lower than default of 2 preparation phase will take a bit longer or if you set higher factor number you might reduce the time. From our experience, value of 2 make usually most of the sense and should not be changed.
  • Support for email document types MSG and EML - Added support for email formats EML and MSG - ReplaceMagic supports changes in hyperlinks in email message body


    Email functionality is part of ReplaceMagic.Ultimate edition.
  • To be able to process documents ReplaceMagic needs access to ...\users\CurrentUser\AppData\Roaming\ReplaceMagic or ...\users\CurrentUser\AppData\Local\ReplaceMagic or ...\ProgramData\ReplaceMagic folder. In case that needed content is not there ReplaceMagic will automatically copy required files from installation folder (usually C:\Program Files (x86)\ReplaceMagic). If this is not possible you will get notification similar to: and scanning or replacement process will exit. It is important to grant permissions to user running ReplaceMagic to be able to create folders and write into them.


  • With parameter "List all skipped documents in Additional info tab" ReplaceMagic will show all documents that are skipped during processing including if document extension was not on list of selected extensions or if document create, last access or last modified dates are not in selected range, etc...
    Previously, ReplaceMagic was showing only skipped documents during real document processing (scenarios like: document is corrupted, document is locked by somebody, document was password protected, etc...).
    To enable this setting go to Configuration => Processing as select "List all skipped documents in Additional info tab":
  • Support for SharePoint not-LegacyAuthProtocols - Added option to select between SharePoint modern and legacy authentication protocols (SharePoint parameter: LegacyAuthProtocolsEnabled):


    In case of this setting on True (default value) you will need to ensure that  SharePoint parameter LegacyAuthProtocolsEnabled is set to true (see ReplaceMagic help article SharePoint MFA (multi-factor authentication). As default setting on False as this setting will work in both cases. If you want you can enable legacy mode by setting it to true. Remark: In case that you have files with % in name when this parameter is set to False ReplaceMagic will not be able to save document after replacements are done and those documents will be shown in Additional Info tab as skipped documents. When legacy mode is set to True ReplaceMagic will not have this constraint but before setting it to True please check that your admins are allowing it. Depending on security policy this parameter might be disabled in your SharePoint environment.
  • Multi-factor authentication added - We added in Beta mode Multi-factor authentication mode to form where you can select where you want to connect on SharePoint side:

    More details about this authentication mode can be found in ReplaceMagic help file under SharePoint MFA (multi-factor authentication)
  • We added additional logging features to measure how time is split during scanning and replacement in documents stored on SharePoint. To enable this option go to Configuration => Logs  and select parameter "SharePoint performance markers"

    After this parameter is enabled in your logs you will see entries like (without comments in bold; btw. this values were when SharePoint was throttling our requests; without throttling we see 4-10 faster performances):
    • Binary Load Document: document.docx: 2143,2041 ms (comment: this is where we download documents from SharePoint)
    • Preparation for Meta Data processing: document.docx: 1270,1174 ms (comment: access to the SharePoint)
    • Replacement process: document.docx: 0,9958 ms(comment: processing done on your computer)
    • CheckIn/Out or Publish/Unpublish update: document.docx: 2007,0992 ms (comment: access to the SharePoint)
    • Preparation for save: document.docx: 89,7238 ms(comment: processing on your computer)
    • Upload to SharePoint: document.docx: 3177,0676 ms (comment: access to the SharePoint)
    • Set document Metadata back: document.docx: 16490,9978 ms (comment: access to the SharePoint)
  • Performance improvements of findMaxID functionality for documents stored in SharePoint
  • Time split when processing documents on SharePoint - Extended result dashboards to include information about processing time split when working with SharePoint. To activate this feature you will need to go to Configuration => Logs and select option "SharePoint performance markers":

    More infos in ReplaceMagic help under: Scanning dashboard & Replacements dashboard
  • Improved error handling if ReplaceMagic databases are not available.
  • Added option to jump to SharePoint configuration directly from form where you can create SharePoint configuration:
  • Included reminder to select only really necessary options for SharePoint processing to avoid unnecessary SharePoint workload.
  • Optimized processing when no SharePoint options are needed to reduce unnecessary calls of SharePoint.
  • ReplaceMagic Themes - from this release you'll have possibility to change look of ReplaceMagic. Under Configuration you can find option "ReplaceMagic Theme"

    Where you can select different design themes for ReplaceMagic installation.
  • SharePoint App-Only authentication mode - SharePoint App-Only authentication mode - we added App-Only Authentication [BETA]

    To use it you will need to create Client ID and Client Secret. Description how to do that can be found in article "Granting access using SharePoint App-Only"
  • Remarks about SharePoint Throttling and current COVID-19 situation => more details in ReplaceMagic help file
  • ReplaceMagic help file reorganized to group most of SharePoint topics under one area
  • Fixed bug that error message was shown when saving ReplaceMagic configuration if ReplaceMagic Theme was not set
  • Changed title "SharePoint Markers" to "SharePoint Metrics"
  • Added option in About form to see validity of trial version
  • Changed default name of Scan Documents export from "Find Documents..." to "Scan Documents"