Yesterday we found a lot of errors in the Project Server queue job history (PWA Settings / Manage Queue Jobs). The Job State was Failed But Not Blocking Correlation, the Job Type was of type Reporting (Project Sync).
To understand the reason, you should know that Project Server synchronizes changes in several lists from the project web sites (PWS) to its reporting database. This kind of synchronization is performed by the Project Server Queue Service (Microsoft.Office.Project.Server.Queuing.exe).
By clicking on the Click to view the error details link we became more information about the background of the problem.
Queue Job Error Details
General
Reporting Wss list sync failed
ReportingWssSyncListFailed (24018) – 101. Details: id=’24018′ name=’ReportingWssSyncListFailed’ uid=’5b39b9cd-bad4-e511-8c5c-005056b45654′ SPListType=’0e2d89a0-e57a-e411-9568-005056b45654′ Error=’101′.
Reporting message processor failed
ReportingWSSSyncMessageFailed (24016) – RDS failed while trying to sync one or more SP lists. The RDS queue message will be retried.. Details: id=’24016′ name=’ReportingWSSSyncMessageFailed’ uid=’6239b9cd-bad4-e511-8c5c-005056b45654′ QueueMessageBody=’ProjectUID=’0e2d89a0-e57a-e411-9568-005056b45654′. ForceFullSync=’False’. SynchronizationType=’Documents” Error=’RDS failed while trying to sync one or more SP lists. The RDS queue message will be retried.’.
Queue
GeneralQueueJobFailed (26000) – ReportingWSSSync.WSSSyncMessageEx. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’ade7bdd3-bad4-e511-8c5c-005056b45654′ JobUID=’485a0869-b8d4-e511-8c5c-005056b45654′ ComputerName=’d30c825e-996d-4a92-a60d-1d66f5340b1b’ GroupType=’ReportingWSSSync’ MessageType=’WSSSyncMessageEx’ MessageId=’1′ Stage=” CorrelationUID=’0fca5f9d-75e8-507f-40b1-bf88f739c79d’. For more details, check the ULS logs on machine d30c825e-996d-4a92-a60d-1d66f5340b1b for entries with JobUID 485a0869-b8d4-e511-8c5c-005056b45654.
The value 101 in the Error property and the value Documents in the SynchronizationType suggest that it is something wrong with the synchronization of the document library list (101 is the ID of the document library list template).
The relevant part from the ULS logs:
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Server General a2tj Medium RowLimit unset…using default RowLimit for query: <Where><Geq><FieldRef Name=’Modified’/><Value Type=’DateTime’ IncludeTimeValue=’TRUE’ StorageTZ=’TRUE’>2016-02-16T14:29:19</Value></Geq></Where> 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x125EC Project Server Project Server Database ah91z Medium Successfully got the connection string (database name=[YourProjectServerDB], id=ae38d4f0-3d16-430b-8cec-c76290a640df, type=Consolidated). Requested access level=ReadWrite: Data Source=YourProjectServerServiceApp;Initial Catalog=YourProjectServerDB;Integrated Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Connect Timeout=15
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation Health 46ri High Throttled:Big list slow query. List item query elapsed time: 0 milliseconds, Additional data (if available): Query HRESULT: 80070024 List internal name, flags, and URL: {7903E344-47BC-42B5-A1BF-14E3E58D817F}, flags=0x0080000000001008, URL="http://YourProjectServer/PWA/Proj1" Current User: 1073741823 Query XML: "<Query><Where><Geq><FieldRef Name="Modified"/><Value Type="DateTime" IncludeTimeValue="TRUE" StorageTZ="TRUE">2016-02-16T14:29:19</Value></Geq></Where></Query>" SQL Query: "N/A" 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General xxpm High Unable to execute query: Error 0x80070024 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General 8kh7 High The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.<nativehr>0x80070024</nativehr><nativestack></nativestack> 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General aix9j High SPRequest.GetListItemDataWithCallback2: UserPrincipalName=i:0).w|s-1-5-21-3634847118-1559816030-2180994487-3302, AppPrincipalName= ,pSqlClient=<null> ,bstrUrl=http://YourProjectServer/PWA/Proj1 ,bstrListName={7903E344-47BC-42B5-A1BF-14E3E58D817F} ,bstrViewName=<null> ,bstrViewXml=<View Scope=’RecursiveAll’><Query><Where><Geq><FieldRef Name=’Modified’/><Value Type=’DateTime’ IncludeTimeValue=’TRUE’ StorageTZ=’TRUE’>2016-02-16T14:29:19</Value></Geq></Where></Query><ViewFields><FieldRef Name=’ID’/><FieldRef Name=’UniqueId’/><FieldRef Name=’FileRef’/><FieldRef Name=’FSObjType’/> ,fSafeArrayFlags=SAFEARRAYFLAG_NONE 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General ai1wu Medium System.Runtime.InteropServices.COMException: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.<nativehr>0x80070024</nativehr><nativestack></nativestack>, StackTrace: at Microsoft.SharePoint.SPListItemCollection.EnsureListItemsData() at Microsoft.SharePoint.SPListItemCollection.get_Count() at Microsoft.Office.Server.Utilities.ContentIterator.ProcessListItems(SPList list, SPQuery query, ItemsProcessor itemsProcessor, ItemsProcessorErrorCallout errorCallout) at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.WSSDataTransfer.BuildIncrementalTransfer(Guid projectUid, SPList splist, SPChangeToken previousChangeToken, WSSListMetadata wssListMetadata, Int32& syncItemsCount) at Microsoft.Office.Project.Ser… 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19* Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General ai1wu Medium …ver.BusinessLayer.ReportingLayer.WSSDataTransfer.BuildTransferData(Guid projectUid, SPList splist, WSSListMetadata wssListMetadata, Boolean forceFullSync, Boolean& syncWasIncremental, Int32& syncItemsCount) at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.TransferWSSListData(Guid projectUid, SPList splist, WSSListMetadata wssListMetadata, DateTime dtStartSyncTime, Boolean forceFullSync, SPChangeToken currentChangeToken) at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.SyncWSSList(Guid projectUid, SPWeb spweb, WSSListMetadata wssListMetadata, Boolean forceFullSync) at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.SyncWSSData(Guid projectUid, SPWeb spweb, WSSTransferMetadata w… 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19* Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General ai1wu Medium …ssTransferMetadata, Boolean forceFullSync) at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.runRDSTransformation(WSSSyncMessageEx wssSyncMessage) at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.HandleMessage(Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext) at Microsoft.Office.Project.Server.BusinessLayer.Queue.ProcessMessageThread.ThreadEntry() at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor.ProcessMessage(Message msg, MessageContext mContext, ProcessMessageThread processMessageThread) at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor.HandleMessages() at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor… 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19* Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General ai1wu Medium ….ThreadEntry() at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor.<>c__DisplayClassb.<Execute>b__a() at Microsoft.Office.Project.Server.Library.SafeCallbackManager.RunCallbackAction(Action action, Func`1 safePredicate) at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor.Execute(Group& nextGroupInCorrelation, JobTicket& nextGroupJobTicket) at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.ProcessMessageGroup(CancellationToken schedulerCancellationToken, Group messageGroup, JobTicket jobTicket, Group& nextGroupInCorrelation, JobTicket& nextGroupJobTicket) at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.ExecuteJobInternal(CancellationToken schedulerCancellationToken, Group& messageGroup) at Microsof… 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19* Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General ai1wu Medium …t.Office.Project.Server.BusinessLayer.Queue.Receiver.<>c__DisplayClass1d.<>c__DisplayClass21.<ExecuteJob>b__1b() at Microsoft.Office.Project.Server.BusinessLayer.Queue.QueueUtils.RunActionWithIgnoreExceptionPredicate(Action action, Predicate`1 ignoreExceptionPredicate, Boolean& isIgnorableException) at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.<>c__DisplayClass1d.<ExecuteJob>b__1a() at Microsoft.Office.Project.Server.Library.SafeCallbackManager.RunCallbackAction(Action action, Func`1 safePredicate) at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.ExecuteJob(CancellationToken schedulerCancellationToken, Int32 siteJobCount) at Microsoft.Office.Project.Server.Services.SiteQueue.<>c__DisplayClass2.<ExecuteJob>b__1() at Microsoft.Office…. 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19* Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General ai1wu Medium …Project.Server.Library.SafeCallbackManager.RunCallbackAction(Action action, Func`1 safePredicate) at Microsoft.Office.Project.Server.Services.SiteQueue.ExecuteJob(CancellationToken ct, Int32 siteJobCount) at Microsoft.Office.Project.Server.Services.QueueJobRunner.ExecuteSiteJob(Object data) at System.Threading.Tasks.Task.Execute() at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot) at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecut… 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19* Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation General ai1wu Medium …ion) at System.Threading.ThreadPoolWorkQueue.Dispatch() 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x16F88 SharePoint Foundation Health 46ri High big list slow query, list:{7903E344-47BC-42B5-A1BF-14E3E58D817F} viewname: viewxml: <View Scope=’RecursiveAll’><Query><Where><Geq><FieldRef Name=’Modified’/><Value Type=’DateTime’ IncludeTimeValue=’TRUE’ StorageTZ=’TRUE’>2016-02-16T14:29:19</Value></Geq></Where></Query><ViewFields><FieldRef Name=’ID’/><FieldRef Name=’UniqueId’/><FieldRef Name=’FileRef’/><FieldRef Name=’FSObjType’/><FieldRef Name=’Author’/><FieldRef Name=’Created’/><FieldRef Name=’Editor’/><FieldRef Name=’Modified’/><FieldRef Name=’Title’/></ViewFields><RowLimit Paged="TRUE">2000</RowLimit></View> 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x16F88 Project Server Reporting auos Warning Standard Information:PSI Entry Point: Project User: PROJECTSERVER\system Correlation Id: 495a0869-b8d4-e511-8c5c-005056b45654 PWA Site URL: http://YourProjectServer/PWA/Proj1 SA Name: ProjectServerApplication PSError: NoError (0) RDS: Problem transferring SharePoint data to the Project Server. Unable to prepare the incremental transfer data for the SharePoint list ‘101’ associated with the project ‘0e2d89a0-e57a-e411-9568-005056b45654’. Error: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.. Full transfer will be used. 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19 Microsoft.Office.Project.Server (0x136D8) 0x16F88 Project Server Reporting auos High PWA:http://YourProjectServer/PWA/Proj1, ServiceApp:ProjectServerApplication, User:PROJECTSERVER\system, PSI: ULS Event: WSSIncrementalTransferFailed, tag 1635086195 was associated with exception: Microsoft.SharePoint.SPQueryThrottledException: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator. —> System.Runtime.InteropServices.COMException: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.<nativehr>0x80070024</nativehr><nativestack></nativestack> at Microsoft.SharePoint.Library.SPRequestInternalClass.GetListItemDataWithCallback2(IListItemSqlClient pSqlClient, String bstrUrl, String bstrListName, String bstrViewName, String bstrViewXml, SAFEARRAYFLAGS fSafeAr… 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19* Microsoft.Office.Project.Server (0x136D8) 0x16F88 Project Server Reporting auos High …rayFlags, ISP2DSafeArrayWriter pSACallback, ISPDataCallback pPagingCallback, ISPDataCallback pPagingPrevCallback, ISPDataCallback pFilterLinkCallback, ISPDataCallback pSchemaCallback, ISPDataCallback pRowCountCallback, Boolean& pbMaximalView) at Microsoft.SharePoint.Library.SPRequest.GetListItemDataWithCallback2(IListItemSqlClient pSqlClient, String bstrUrl, String bstrListName, String bstrViewName, String bstrViewXml, SAFEARRAYFLAGS fSafeArrayFlags, ISP2DSafeArrayWriter pSACallback, ISPDataCallback pPagingCallback, ISPDataCallback pPagingPrevCallback, ISPDataCallback pFilterLinkCallback, ISPDataCallback pSchemaCallback, ISPDataCallback pRowCountCallback, Boolean& pbMaximalView) — End of inner exception stack trace — at Microsoft.SharePoint.SPGlobal.HandleThrottleException(… 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19* Microsoft.Office.Project.Server (0x136D8) 0x16F88 Project Server Reporting auos High …COMException comEx) at Microsoft.SharePoint.Library.SPRequest.GetListItemDataWithCallback2(IListItemSqlClient pSqlClient, String bstrUrl, String bstrListName, String bstrViewName, String bstrViewXml, SAFEARRAYFLAGS fSafeArrayFlags, ISP2DSafeArrayWriter pSACallback, ISPDataCallback pPagingCallback, ISPDataCallback pPagingPrevCallback, ISPDataCallback pFilterLinkCallback, ISPDataCallback pSchemaCallback, ISPDataCallback pRowCountCallback, Boolean& pbMaximalView) at Microsoft.SharePoint.SPListItemCollection.EnsureListItemsData() at Microsoft.SharePoint.SPListItemCollection.get_Count() at Microsoft.Office.Server.Utilities.ContentIterator.ProcessListItems(SPList list, SPQuery query, ItemsProcessor itemsProcessor, ItemsProcessorErrorCallout errorCallout) at Microsoft.Office.P… 0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19* Microsoft.Office.Project.Server (0x136D8) 0x16F88 Project Server Reporting auos High …roject.Server.BusinessLayer.ReportingLayer.WSSDataTransfer.BuildIncrementalTransfer(Guid projectUid, SPList splist, SPChangeToken previousChangeToken, WSSListMetadata wssListMetadata, Int32& syncItemsCount) at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.WSSDataTransfer.BuildTransferData(Guid projectUid, SPList splist, WSSListMetadata wssListMetadata, Boolean forceFullSync, Boolean& syncWasIncremental, Int32& syncItemsCount), LogLevelManager Warning-ulsID:0x61756F73 has no entities explicitly specified. 0fca5f9d-75e8-507f-40b1-bf88f739c79d
The most important message from the logs:
The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator
That means, the value set for the List View Threshold (configurable in Resource Throttling in Web Application Management site of the Central Administration, defaults to 5000) is less than the document count in the library. You should increase the value to re-enable synchronization, using PowerShell, for example:
$webApp = Get-SPWebApplication http://YourProjectServer
$webApp.MaxItemsPerThrottledOperation = 10000
$webApp.Update()
Or alternatively convince your users not to store so many files in the document library.
