What is new in 2023.2:
- Fixed problem when on DFS share processing is stopping after first folder found with Access Denied permissions.
As outside of ReplaceMagic components are used for creation of list of files for scanning, we cannot show skipped folders directly in ReplaceMagic but AccessDeniedLocations.log file will be created in c:\users\username\appdata\roaming\replacemagic\logs where you can find what was skipped. If no file is found that means that there were no locations with Access Denied. Failback location is c:\Temp.
- Improved check of relative links with documents in SharePoint.
- Extended filtering of temporary files (starting with ~$) uploaded to the SharePoint.
- Added option to remove parameters from URL which is useful in case of migrations to SharePoint when there are URLs like: https://old.system.com/sites/newsitename/path/to/file/somefile.xlsx?d=w152a583596ef4cc198abe15ad350117d as after migration parameter d will point to old document ID which will not work anymore. To set parameters for removal use Configuration => Misc:
What is new?
With this configuration ReplaceMagic will remove d and p parameters from URL. Separator between parameters is | and this will work only for links in Hyperlinks section. Replacement is not case-sensitive so if you specify d=, ReplaceMagic will replace parameter d= but also D=.
Prerequisite: you will need to provide search&replace strings as otherwise replacement process cannot start and changes in hyperlinks need to be selected.
- Further reduction of requests to SharePoint to reduce workload triggered by ReplaceMagic.
- Fixed bug that in rare cases under heavy throttling ReplaceMagic can close.
- In SharePoint article https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online (last update from 03/20/2023; page is regularly updated) Microsoft is stating that beside user throttling there will be application throttling also. Every application will have own limits per tenant no mater if Microsoft Graph, CSOM or REST is used (ReplaceMagic is using CSOM and REST). Limits are as following (taken from MSFT site):
Also different resource unit costs per request are defined:
(example is for Microsoft Graph, where CSOM and REST are even more "expensive"). To translate to ReplaceMagic terminology - scanning is "easy" request as we need only to download file but in case of replacements multiple requests are required for:
- Meta data preparation
- Download file
- Upload changed file
- Optionally, set meta data back (last modification date, editor, status of file, version...)
This will cause that replacement will require more than few request which will be taken from your quota. To reduce risk of throttling, we are strongly recommending not to use more than 2 documents in parallel (default is 4) as there will be great chance that you will be throttling. In case of throttling ReplaceMagic will automatically reduce parallelism to 1 (sequential processing) and if parameter Optimize workload is selected ReplaceMagic will start to increase parallelism after defined number of processed documents (new value 100 (reduced from 500) without throttling.
According to Microsoft documentation, when throttling is happening there has to be used Retry-After parameter which will stop processing for SharePoint requested duration value.
- Added parameter "Use SharePoint retry-after recommendation" - when SharePoint throttles you, it will send duration during which processing cannot continue. By using this parameter, ReplaceMagic will stop processing for SharePoint requested period and continue afterwards. Default: selected.
- In case that you do not want to use SharePoint retry-after parameters values via parameter "Throttling retry-after period" you can define own wait period (default: 120). This parameter is only visible if parameter "Use SharePoint retry-after recommendation" is unchecked:
Due to this parameter when ReplaceMagic is first time throttled, there will be pause of 120 seconds before processing can continue and you will see that information in progress bar. Using this and above parameter we are protecting your processing to be totally stopped and to have as less skipped documents due to throttling as possible.
What is new:
- WebLogon authentication used Microsoft Browser component to log or get logon data for SharePoint. As this component is based on Internet Explorer, it will start not work for SharePoint Modern sites logon as Microsoft is retiring Internet Explorer. To address this, we changed that ReplaceMagic is using new WebView2 component based on Chromium which supports modern sites also.
As this required installation of Microsoft WebView2 component, ReplaceMagic setup will trigger deployment if installation is not present.
WebViev component will store cache in c:\users\username\appdata\roaming\replacemagic\EBWebView which might grow with the time. If this folder is deleted, you will need to login again. In case that command line is used, please ensure that you first login via ReplaceMagic interfaces as only then cache entries are present and logon will work without user interaction.
- New parameter added to empty WebView2 cache on ReplaceMagic exist. Per default, parameter is not set.
- Fixed bug that changes are not done in Text and Formulas in Excel files.
- Name Manager - included that ReplaceMagic is not making changes of Name Manager links if they are longer than 235 characters to avoid hitting Excel limits as that will remove Name Manager links.
- Name Manager - include processing only of Name Manager references which are visible in Excel (as there might be some hidden Name Manager references not visible to users)
- Name Manager - as there might be a lot of name manager links which are mainly internal, per default, ReplaceMagic will not include them. In case that you expect external links in Name Manager links select option "Include Name Manager links".
Expect that this will include a lot of link in Link Source section where majority (if not all) will be internal links that you can ignore.
- Processing of huge documents (>250Mb) on SharePoint.Online - Per default SharePoint.Online does not allow upload of documents larger than 250 Mb which means that ReplaceMagic will report error and documents will not be changed. To allow huge documents upload, ReplaceMagic will split document in blocks of 245Mb allowing upload of any documents size (ok, there might be some other SharePoint.Online limits like maximum file size but this is different story). This will work only if you allow processing of larger documents (parameter in Configuration => Configuration => Skip files larger than) as per default ReplaceMagic is skipping documents larger than 100Mb.
Constraints:
- This will not work in case of App-Only authentication [beta]
- In case that upload is, for any reason, interrupted before it is completed (all blocks), files will be damaged as only part of file is uploaded. Btw. same will happen also with any other tool uploading documents in blocks.
- Added support for ReplacePreview mode for list item links.
- Extended log file creation to include application version.
- Changed setting name "Update Read Folder status" to "Refresh preparation status".
- Extended "Refresh preparation status" to include progress bar updates when filtering documents.
- Extended processing of FieldCodes of type FieldHyperlink and property Address and SubAddress.
- As WebView cache can grow we added button to clear it manually.
- Improved processing of multi language sites.
- Improved PowerQueries processing.
- New theme added: VS2022.
What is new (2022.4):
- Fixed replacement of SharePoint page hyperlinks containing in URL & - problem was that only first URL was changed
- Web Logon authentication mode set as default for SharePoint login
- Extended log capturing in case of SharePoint throttling
- Fixed problem that SharePoint lists where name is starting after letter U were not visible
- Convert spaces in link sources to %20 to mimic same way how Excel is saving them. This is important as that will have direct impact on link source length which was limit of 255 characters
- Fixed Visio hyperlinks replacements bug that ExtraInfo (link part after ? is not saved). This was happening if initial link did not have ExtraInfo but new link has
- ReplaceMagic now supports changes in Name Manager link. To do that select that changes should be in Link Sources which included Name Manager also
- Extension of ReplaceMagic to support document encrypted by Microsoft Information Protection. Initial focus on Excel, Word and PowerPoint documents
- For Microsoft Information Protection we need latest version of Microsoft Visual C++ Redistributable Visual Studio 2015, 2017, 2019, and 2022 (direct download link) which is also included in installation file
- Fixed bug that PDF documents might stay checked-out after processing on SharePoint
- Improved processing of data connections in Excel files
- Further improvements of processing SharePoint Canvas pages
- If SharePoint page has Canvas or Wiki content (page content stored in SharePoint database) changes of links there will have priority to hard-coded links. In general, there should not be hard-coded pages (page content not stored in SharePoint database)
- Extended functionality that digitally signed documents (PDF, Excel, PowerPoint or Word) can be skipped during processing
What is new in version 2022.3:
- Memory management improved - memory leak found and fixed when having a lot of documents skipped due to timeout.
Comment: in case of skipped documents - findings from them (hyperlinks, OLE, LinkSources...) might appear in processing result grids as document processing was canceled after going through document structure. Unfortunately, there is no way to remove those results for the time being.
- What is new (2022.3) Replacements in PDF documents simplified so there is only selection "Hyperlinks" instead "Hyperlinks" and "Launch Actions"
- Previously, in case of major/minor enabled document libraries if active document was in major version ReplaceMagic was unpublishing it and then republishing. This is removed now in case if you set that you need versioning but Major or Minor version. In case that you need Keep Version Number unpublish will be still required as otherwise SharePoint will not allow change (error: Version conflict).
- Maximum length of formulas was extended to 8192 characters. Changes where formula is longer than 8192 characters will be stopped to avoid document corruption. Side comment: to fix formula focus on Link Sources as after fixing Link Source(s) Excel will automatically update formulas.
- Scanning documents export default option changed to most complete report
- In rare cases, dependent on the tool used to create PDF documents, replacement in LinkAnnotation objects of PDF documents (hyperlinks) was making links unclickable which is fixed now.
What is new (2022.2)
- Improved functionality to keep last modification date in case of changes in SharePoint. Focus on updates when SharePoint site having different time zone to user running ReplaceMagic
- Extended command line run configuration that type of scanning can be also set (full scan results or ultra fast scan)
- Extended import of skipped documents to show import progress
- Fixed Link Sources check broken link counters for hidden Link Sources
- Extended processing of SharePoint pages with hard coded text and regular expressions
- Fix bug that changes in hard-coded SharePoint pages did not work properly
New in version 2022.1 (16th of January, 2022)
- Extended log file and Console view to report which document areas were selected for replacement process.
- Extended that if ReplaceMagic is executed with silent run parameters file complete configuration is taken over so if ReplaceMagic is used to process documents manually after silent run (for example, reprocessing of skipped documents) same parameters including search&replace strings are used.
- Extended CommandLine parameters file with 2 undocumented parameters that can be added before closing XML tag </COMMANLINEPARAMETERS>:
Those 2 parameters will overwrite document processing timeout parameters in ReplaceMagic configuration file and have to be added manually directly into silent run file.
- If in form to add SharePoint site button Test was pressed and then form was closed pressing X in upper right corner, ReplaceMagic was showing error popup - problem resolved now.
- In rare occasions, when multiple users are used to connect to the SharePoint, during document processing, SharePoint client context can be wrongly set causing that documents are skipped (they'll appear in Skipped Documents tabs where they can be reprocessed again. Error message will be something like "The object is used in the context different from the one associated with the object.").
- Fixed problem that Power Queries in documents saved with non-English Excel are not recognized.
- Fixed problem that Variation Labels list was not skipped in all scenarios.
- Fixed problem that documents with extension in upper case were skipped.
- Extended error tracking when creating list of documents in SharePoint (throttling, error 429 or 503)
- Fixed bug that only one change in OneNote text area can be done.
- Fixed bug "Enforce check-in [checked-out docs will be checked-in for processing]" stopped to work after 2021.3. Also changed option title to "Try to change checked-out document(s)"
- Minimum .NET version changed to 4.7.2 as previously used 4.6.1 will be retired in May, 2022.
- Fixed bug that in Ultra fast scan links of Quick Launch and Top Nav Bar were shown (time was spent on reading them).
- Fixed bug that workload management (decrease/increase number of parallel documents for processing) did not work when throttling message did not contain (429) or (503).
- In case that SharePoint DenyAddAndCustomizePages is enabled (meaning that you do not have permissions to make changes in SharePoint pages)
ReplaceMagic will now report "Access denied (check ReplaceMagic help - Processing of SharePoint .aspx pages stopped working"):
Previously ReplaceMagic will report problem to save page without conclusive explanation why.
- If option "Remove Link Sources from Excel docs" is used (in Search & Replace configuration form) -> after removing external references formulas will be also updated in form => If formula was “=‘SomeExcelFile.xlam’!customfunction()”, after removing the external link “SomeExcelFile.xlam”, this cell’s formula will become “=customfunction().
-
Trial limitation and ReplacePreview mode - main trial limitation is that changes in 10% of scanned documents will be done. To get impression of complete replacement process from this version if you select "Replace Preview" option in Search&Replace form, during testing in trial mode, ReplaceMagic will report all changes in scanned documents according your entered search & replace strings. As this is Replace Preview mode no changes will be saved but you will get list of all what will be replaced in case that you purchase ReplaceMagic license.
New in version 2021.5
- Added option that ReplaceMagic removes all link sources from your Excel documents:
- What is new (2021.5)?
Extended command line run to include filtering of files based on dates:
- Changed that per default ReplaceMagic will not check-out/check-in documents when processing files on SharePoint. Reason is that if this is done and there is SharePoint throttling which will cause that files are skipped, documents might stay checked-out. In case that you want to to keep publishing or approval status you will need to have this feature enabled.
- Fixed bug that after SharePoint pages replacements are done published pages are staying in unpublished state (minor version).
- Sorting of search & replace strings - As sort order of search&replace strings is very import ReplaceMagic will now sort them automatically according to their length descending when you press button to start replacement process or to prepare parameters for command line run. This is required as if you have shorter strings first ReplaceMagic might make changes when shorter string is found and as it is also part of longer search string that might cause that longer string is skipped as changes are already done. This is done in the background to ensure that strings are properly sorted but sort order will not be visible in search and replace form.
As a indicator, we added new column "Search length" which can be used as indicator to show length of your search strings
- Extended ReplaceMagic to automatically take care of approval and/or publishing status of documents stored in SharePoint when this setup is used. Previously, user had to select what ReplaceMagic should do.
New in version 2021.4
- There is a chance that Link Sources in Excel files are not visible if you go to Data=>Connection=>Edit Links but that they are still present if you check XML document structure - to support analyses on this side ReplaceMagic will include in Link Source tab additional field called VisibleInExcel which will help to better understand structure of link source stored in documents.
In case that this field is not visible go to c:\users\username\appdata\roaming\replacemagic or c:\users\username\appdata\local\replacemagic and delete file gridLinkSourcesLayout.xml which will reset result grid fields (btw. this will show default fields order).
- Added option that ReplaceMagic during replacements in hyperlinks, OLE, Link Sources, Pivot or Power Queries makes replacement of special characters. This is useful in case of documents stored in SharePoint as some migration tools are converting locations of copied documents by replacing special characters.
Option "Use illegal characters conversion during check of broken links?" will instruct ReplaceMagic to replace special characters when links are checked. This will not make any changes of links.
Option "Replace SharePoint illegal characters?" will instruct ReplaceMagic to convert illegal characters during replacement process similar to conversion of /and \.
Special characters are converted in order from 1 to 18 and empty fields are skipped.
- Extended processing of SharePoint .aspx page where content is part directly of the page. ReplaceMagic will read all tags used in .aspx page and make replacement of them. Previously, only content between <mso:PublishingPageContent></mso:PublishingPageContent> was processed (default SharePoint pages).
- Fixed bug that silent run processing replacement process did not work.
- Added parameter to control if ReplaceMagic will automatically close after silent run processing is finished.
- Fixed bug that when document processing timeout was reached number of active threads was not reduced after closing processed document which could lead that processing is stopped waiting on free processes.
- Changed that ReplaceMagic in command line mode is closing splash screen before processing documents.
This time something different :)
For some reason since weeks I had problem to connect to O365 mailbox. Initially, it stopped work so I decided to remove mailbox from Outlook (2016) and to add it again but that did not work. For whatever reasons, no matter, what I tried Outlook was not able to connect to O365.
Yesterday, it started to be even worse as when I went to SharePoint Online to test processing of documents all was ok and when I tried to open documents in Excel Online that worked too but opening of files through normal version of Excel was not working.
Surprisingly, Excel was telling me that I am already logged from some other machine which was not the case and additionally any Office desktop application was showing error next to username.
By opening this I saw: "Error - Sorry we can't get to your account right now. To fix this please sign in again." and logically, I tried to sign in again but that did not work so I was thinking "great, now beside Outlook, rest of Office apps are not working".
Of course, I did sign-out and then again sign-in but no progress. I even checked my user account under Windows Settings => Access work or school but that did not help.
Ultimately, solution was to go to Office app (I tried Excel) and to sign-out and then to go to Windows Settings => Access work or school where I clicked on Disconnect below my O365 account. After doing restart (not sure if this is really needed) I went to Excel and this time I logged again without any warnings and then I tried to add again my O365 account to Outlook 2016 and that worked.
What is new in ReplaceMagic 2021.3:
- SharePoint Online throttling - Recently, we noticed that Microsoft is even more protecting SharePoint Online platform by slowing down processing of documents.
As during scanning and replacements ReplaceMagic needs to download documents, read document meta data, apply check-in/check-out settings, keep status of documents (approval, publishing) and finally to upload documents and set all meta data back to normal you can see performance degradation. To avoid this move document processing outside of business hours and do not over parallelize processing of documents as otherwise duration of processing can be impacted. Of course, we will try to protect processing but throttling, which is normal to expect, is, unfortunately, outside of our hands.
Additional infos can be found in article SharePoint Throttling
In case of skipped documents during replacement phase - Due to those limitations some documents might be skipped but ReplaceMagic offers option to import skipped files and process them after main replacement phase is done. To do that, in newly, shown tab Additional Info click on:
This can be repeated until not all documents are not processed.
We would also recommend to split processing on few smaller batches/locations to avoid having processing against large collection of documents at once.
- SharePoint performance metrics changed to show average processing time per document as previously ReplaceMagic was showing total phase processing time. Average processing time is calculated as total phase time / total processed documents. Those metrics are part of result dashboards.
- SharePoint Web Logon- sometimes Web Logon authentication might take longer. If this is taking more than 30+ in newly opened popup click on button:
In case that SharePoint structure is not shown in main SharePoint form please close it and open it again (we are working on this rare refresh problem).
- Added functionality to create Windows registry entries for creation of Windows dumps if ReplaceMagic is crashing. This can be done over Configuration => Logs
This will work only if your user has enough Windows permissions.
Dump files are required for root-cause analyses by ReplaceMagic team and per default they will be stored in C:\Users\USERNAME\Local\ReplaceMagic\CrashDumps.
Depending on your Windows policies this might not work but we recommend set it anyway.
IMPORTANT: Do this only on request for ReplaceMagic team
- Fix bug that in some cases documents on local locations (disks, file servers,...) were skipped.
Check only XML stored Link Sources - Excel link sources are stored in XML structure under /xl/externalLinks/_rels location but during check if links are broken ReplaceMagic will read this location but also location which is used in Excel if you go to Data => Edit links. To avoid false-positive results, we include feature to disable check of link which is shown in Excel:
Per default we are checking both location and if one is working ReplaceMagic will report that link is ok.
- In case that Excel document is moved to new location hard-coded AbsolutePath value will still point to the old location. Absolute path value is used to show link sources if you open Excel document and go to Data => Edit Link. From this version, in case that there are Link Source changes ReplaceMagic will update Absolute Link to point to new location where documents is. This means, when you open Excel file and go to Data => Edit Links you will see link pointing to some URL or drive/network location like where document is stored.
Fixed bug that was triggering "expired maintenance" when maintenance was still valid.
"Native SharePoint check of broken links" - extended support for additional case of hyperlinks in list items (field of URL type). Keep in mind that this feature will send additional requests to SharePoint what might increase risk to get throttled by SharePoint.
Added new tab under Configuration for misc features - "Convert relative to absolute links" used to define how relative links will be checked. Per default when ReplaceMagic finds relative link, it will merge that link and location of file containing this link which might be misleading in some cases. With this feature you can specify how link should start and what should be prefix but also if slashes has to be converted:
Additionally, ReplaceMagic can write to log file original address and also address where prefix was added for your reference.
Keep in that this will have performance impact as due to, probably, large number of links ReplaceMagic will need to write a lot to log files. Of course, this is only active during scanning process.
Remark: original links will not be impacted by this as this change is used only for check of links without making real changes.
- Started to add functionality to process SharePoint Quick Launch and Top Menu Bar.
IMPORTANT: In case that SharePoint cannot recognize new menu link as valid, change will not be saved. Keep in mind that you have to convert menu link to valid link.
Log files should be now created after any of processing activities (scanning documents or replacement process). Log files are per default saved in c:\users\username\appdata\roaming\replacemagic\logs or in c:\users\username\appdata\local\replacemagic\logs and filename format is: ReplaceMagic - Datum - Time.log (for example, ReplaceMagic - 13.05.2021 135259.log)
Fix bug that scanning of .aspx pages (in SharePoint) was reporting more links than pages really have.
Added encoding/decoding of additional special characters in .aspx pages.
Extended convert slash/backslash functionality to changes in Excel text area (convert functionality will be applied only on changed cell content).
Formatting of SharePoint labels in scanning or replacement dashboard adjusted => (SP) moved to second row.
- Added parameter "Fast Calendar Step" which can be used to set how many months ReplaceMagic calendar will move to the left or right when << or >> are pressed:
- Added option that ReplaceMagic converts Unicode characters in SharePoint/Web pages:
