Suddenly getting UserNotFoundException: User Not Found: Could not load profile data from the database

Starting yesterday (I believe) we are suddenly receiving a "User not found" error when trying to view a user's profile page when clicking on their name (in search or elsewhere) via the url
https://www.contoso.org/Person.aspx?accountname=contoso%255Cmyuserlogin
This is happening for all users. (but profiles exist and seem ok, and sync with AD is running fine) When looking in the logs I see the error  (full ULS log is later below)
Exception in LoadRequestedUserProfile: (Microsoft.Office.Server.UserProfiles.UserNotFoundException: User Not Found: Could not load profile data from the database.
The only clue of what MAY be going on is just before that I see the following two lines which came up after I changed the Database logging to Verbose:
ConnectionString: 'Data Source=INSTANCE;Initial Catalog=UPAPROFILEDB;Integrated Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Connect Timeout=15;Application Name=SharePoint[w3wp][3][UPAPROFILEDB]'    Partition:
NULL ConnectionState: Closed ConnectionTimeout: 15
SQL connection time: 0.0456 for Data Source=INSTANCE;Initial Catalog=UPAPROFILEDB;Integrated Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Connect Timeout=15;Application Name=SharePoint[w3wp][3][HNet3_Svc_Prod_UPA_Profile] 
If I am reading that correctly, does it mean the connection to the database is timing out instantly? I am able to use powershell to pull up profiles and print them to the screen while on the webservers. And it's only the one specific page that is not
working.
What sort of tests or further debugging can I do for the error above?
We are on premise, SharePoint 2013 sp1, sql2012, server 2012, claims using the full SharePoint Profiile Sychronization against AD.  Farm has been running since last august. Last week the admin team did run windows update on all servers, I can't be sure
if it was broken for 5 days and no one noticed or if it broke yesterday.
I already did the following:
Verified the profiles do exist in central admin.
I can still access their personal sites directly via
https://www.contoso.org/personal/myuserlogin
I checked the ProfileDB database and made sure the service accounts for the UPA has dbowner, so does the farm account, and the web application pool has SPDataAccess.
Verified the AD sync account has proper rights in AD.
I reran a full synchronization which ran with no errors.
File systems on the webservers and appserver have plenty of free space.
Rebooted every server in farm
I even checked the bin directory of the mysite webapplication, (since once I had a weird problem where access to bin was removed.) WSS_ADMIN_WPG, WSS_RESTRICTED_WPG_W4 have full control and WSS_WPG,IIS_IUSRS all have read and execute.
I opened powershell on the webserver I can indeed load profiles and print their values.
Timestamp Area Category EventID Level Message Correlation
07/25/2014 14:11:08.43 SharePoint Foundation Monitoring nasq Medium Entering monitored scope (Request (GET:https://my.hungernet.org:443/Person.aspx?accountname=FA%255Ccbuchholz)). Parent No
07/25/2014 14:11:08.43 SharePoint Foundation Logging Correlation Data xmnv Medium Name=Request (GET:https://my.hungernet.org:443/Person.aspx?accountname=FA%255Ccbuchholz) be11a89c-fbe8-a08d-57d2-afa6a4948743
07/25/2014 14:11:08.43 SharePoint Foundation Micro Trace uls4 Medium Micro Trace Tags: 0 nasq be11a89c-fbe8-a08d-57d2-afa6a4948743
07/25/2014 14:11:08.43 SharePoint Foundation Monitoring b4ly Medium Leaving Monitored Scope (Request (GET:https://my.hungernet.org:443/Person.aspx?accountname=FA%255Ccbuchholz)). Execution Time=1.3859 be11a89c-fbe8-a08d-57d2-afa6a4948743
07/25/2014 14:11:08.47 SharePoint Foundation Monitoring nasq Medium Entering monitored scope (Request (GET:https://my.hungernet.org:443/Person.aspx?accountname=FA%255Ccbuchholz)). Parent No
07/25/2014 14:11:08.47 SharePoint Foundation Logging Correlation Data xmnv Medium Name=Request (GET:https://my.hungernet.org:443/Person.aspx?accountname=FA%255Ccbuchholz) be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.49 SharePoint Foundation General ajji6 High Unable to write SPDistributedCache call usage entry. be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50 SharePoint Foundation Authentication Authorization agb9s Medium Non-OAuth request. IsAuthenticated=True, UserIdentityName=0#.w|fa\cbuchholz, ClaimsCount=167 be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50 SharePoint Foundation Files ak8dj High UserAgent not available, file operations may not be optimized. at Microsoft.SharePoint.SPFileStreamManager.CreateCobaltStreamContainer(SPFileStreamStore spfs, ILockBytes ilb, Boolean copyOnFirstWrite, Boolean disposeIlb) at Microsoft.SharePoint.SPFileStreamManager.SetInputLockBytes(SPFileInfo& fileInfo, SqlSession session, PrefetchResult prefetchResult) at Microsoft.SharePoint.CoordinatedStreamBuffer.SPCoordinatedStreamBufferFactory.CreateFromDocumentRowset(Guid databaseId, SqlSession session, SPFileStreamManager spfstm, Object[] metadataRow, SPRowset contentRowset, SPDocumentBindRequest& dbreq, SPDocumentBindResults& dbres) at Microsoft.SharePoint.SPSqlClient.GetDocumentContentRow(Int32 rowOrd, Object ospFileStmMgr, SPDocumentBindRequest& dbreq, SPDocumentBindResults& dbres... be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50* SharePoint Foundation Files ak8dj High ...) at Microsoft.SharePoint.Library.SPRequestInternalClass.GetFileAndMetaInfo(String bstrUrl, Byte bPageView, Byte bPageMode, Byte bGetBuildDependencySet, String bstrCurrentFolderUrl, Int32 iRequestVersion, Byte bMainFileRequest, Boolean& pbCanCustomizePages, Boolean& pbCanPersonalizeWebParts, Boolean& pbCanAddDeleteWebParts, Boolean& pbGhostedDocument, Boolean& pbDefaultToPersonal, Boolean& pbIsWebWelcomePage, String& pbstrSiteRoot, Guid& pgSiteId, UInt32& pdwVersion, String& pbstrTimeLastModified, String& pbstrContent, UInt32& pdwPartCount, Object& pvarMetaData, Object& pvarMultipleMeetingDoclibRootFolders, String& pbstrRedirectUrl, Boolean& pbObjectIsList, Guid& pgListId, UInt32& pdwItemId, Int64& pllListFlags, Boolean& pbAccessDenied, Guid& pgDocid, Byte& piLevel, UInt64& ppermMask, ... be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50* SharePoint Foundation Files ak8dj High ...Object& pvarBuildDependencySet, UInt32& pdwNumBuildDependencies, Object& pvarBuildDependencies, String& pbstrFolderUrl, String& pbstrContentTypeOrder, Guid& pgDocScopeId) at Microsoft.SharePoint.Library.SPRequestInternalClass.GetFileAndMetaInfo(String bstrUrl, Byte bPageView, Byte bPageMode, Byte bGetBuildDependencySet, String bstrCurrentFolderUrl, Int32 iRequestVersion, Byte bMainFileRequest, Boolean& pbCanCustomizePages, Boolean& pbCanPersonalizeWebParts, Boolean& pbCanAddDeleteWebParts, Boolean& pbGhostedDocument, Boolean& pbDefaultToPersonal, Boolean& pbIsWebWelcomePage, String& pbstrSiteRoot, Guid& pgSiteId, UInt32& pdwVersion, String& pbstrTimeLastModified, String& pbstrContent, UInt32& pdwPartCount, Object& pvarMetaData, Object& pvarMultipleMeetingDoclibRootFolders, String& pbst... be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50* SharePoint Foundation Files ak8dj High ...rRedirectUrl, Boolean& pbObjectIsList, Guid& pgListId, UInt32& pdwItemId, Int64& pllListFlags, Boolean& pbAccessDenied, Guid& pgDocid, Byte& piLevel, UInt64& ppermMask, Object& pvarBuildDependencySet, UInt32& pdwNumBuildDependencies, Object& pvarBuildDependencies, String& pbstrFolderUrl, String& pbstrContentTypeOrder, Guid& pgDocScopeId) at Microsoft.SharePoint.Library.SPRequest.GetFileAndMetaInfo(String bstrUrl, Byte bPageView, Byte bPageMode, Byte bGetBuildDependencySet, String bstrCurrentFolderUrl, Int32 iRequestVersion, Byte bMainFileRequest, Boolean& pbCanCustomizePages, Boolean& pbCanPersonalizeWebParts, Boolean& pbCanAddDeleteWebParts, Boolean& pbGhostedDocument, Boolean& pbDefaultToPersonal, Boolean& pbIsWebWelcomePage, String& pbstrSiteRoot, Guid& pgSiteId, UInt32& pdwVersion,... be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50* SharePoint Foundation Files ak8dj High ... String& pbstrTimeLastModified, String& pbstrContent, UInt32& pdwPartCount, Object& pvarMetaData, Object& pvarMultipleMeetingDoclibRootFolders, String& pbstrRedirectUrl, Boolean& pbObjectIsList, Guid& pgListId, UInt32& pdwItemId, Int64& pllListFlags, Boolean& pbAccessDenied, Guid& pgDocid, Byte& piLevel, UInt64& ppermMask, Object& pvarBuildDependencySet, UInt32& pdwNumBuildDependencies, Object& pvarBuildDependencies, String& pbstrFolderUrl, String& pbstrContentTypeOrder, Guid& pgDocScopeId) at Microsoft.SharePoint.SPWeb.GetWebPartPageContent(Uri pageUrl, Int32 pageVersion, PageView requestedView, HttpContext context, Boolean forRender, Boolean includeHidden, Boolean mainFileRequest, Boolean fetchDependencyInformation, Boolean& ghostedPage, String& siteRoot, Guid& siteId, Int64& bytes, ... be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50* SharePoint Foundation Files ak8dj High ...Guid& docId, UInt32& docVersion, String& timeLastModified, Byte& level, Object& buildDependencySetData, UInt32& dependencyCount, Object& buildDependencies, SPWebPartCollectionInitialState& initialState, Object& oMultipleMeetingDoclibRootFolders, String& redirectUrl, Boolean& ObjectIsList, Guid& listId) at Microsoft.SharePoint.ApplicationRuntime.SPRequestModuleData.FetchWebPartPageInformationForInit(HttpContext context, SPWeb spweb, Boolean mainFileRequest, String path, Boolean impersonate, Boolean& isAppWeb, Boolean& fGhostedPage, Guid& docId, UInt32& docVersion, String& timeLastModified, SPFileLevel& spLevel, String& masterPageUrl, String& customMasterPageUrl, String& webUrl, String& siteUrl, Guid& siteId, Object& buildDependencySetData, SPWebPartCollectionInitialState& initialState, ... be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50* SharePoint Foundation Files ak8dj High ...String& siteRoot, String& redirectUrl, Object& oMultipleMeetingDoclibRootFolders, Boolean& objectIsList, Guid& listId, Int64& bytes) at Microsoft.SharePoint.ApplicationRuntime.SPRequestModuleData.GetFileForRequest(HttpContext context, SPWeb web, Boolean exclusion, String virtualPath) at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.InitContextWeb(HttpContext context, SPWeb web) at Microsoft.SharePoint.WebControls.SPControl.SPWebEnsureSPControl(HttpContext context) at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.GetContextWeb(HttpContext context) at Microsoft.SharePoint.ApplicationRuntime.SPRequestModule.PostResolveRequestCacheHandler(Object oSender, EventArgs ea) at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IEx... be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50* SharePoint Foundation Files ak8dj High ...ecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error) at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb) at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompl... be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50* SharePoint Foundation Files ak8dj High ...etion(IntPtr pHandler, RequestNotificationStatus& notificationStatus) at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50 SharePoint Foundation Files aiv4w Medium Spent 0 ms to bind 4224 byte file stream be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50 SharePoint Foundation Logging Correlation Data xmnv Medium Site=/ be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.50 SharePoint Foundation Monitoring b4ly High Leaving Monitored Scope (PostResolveRequestCacheHandler). Execution Time=11.7341 be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.52 SharePoint Portal Server User Profiles aj2aw Verbose GetPartitionPropertiesCache :: ApplicationId = c604ddac-b9f9-4661-b31f-44cdcf2b78dc be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.52 SharePoint Portal Server User Profiles aj2ax Verbose GetPartitionPropertiesCache :: Trying to find existing cache be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.52 SharePoint Portal Server User Profiles aj2az Verbose GetPartitionPropertiesCache :: Getting Cached object be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.54 SharePoint Portal Server User Profiles aj2aw Verbose GetPartitionPropertiesCache :: ApplicationId = c604ddac-b9f9-4661-b31f-44cdcf2b78dc be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.54 SharePoint Portal Server User Profiles aj2ax Verbose GetPartitionPropertiesCache :: Trying to find existing cache be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.54 SharePoint Portal Server User Profiles aj2ay Verbose GetPartitionPropertiesCache :: Found existing cache in httpcontext be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.54 SharePoint Server Database tzku Verbose ConnectionString: 'Data Source=HungerNet3;Initial Catalog=HNet3_Svc_Prod_UPA_Profile;Integrated Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Connect Timeout=15;Application Name=SharePoint[w3wp][3][HNet3_Svc_Prod_UPA_Profile]' Partition: NULL ConnectionState: Closed ConnectionTimeout: 15 be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.54 SharePoint Server Database ahjqp Verbose SQL connection time: 0.0583 for Data Source=HungerNet3;Initial Catalog=HNet3_Svc_Prod_UPA_Profile;Integrated Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Connect Timeout=15;Application Name=SharePoint[w3wp][3][HNet3_Svc_Prod_UPA_Profile] be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.54 SharePoint Foundation General adyrv High Cannot find site lookup info for request Uri http://www.hungernet.org/. be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.54 SharePoint Foundation General adyrv High Cannot find site lookup info for request Uri http://www.hungernet.org/. be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.54 SharePoint Foundation General adyrv High Cannot find site lookup info for request Uri http://lnkd.in/bb6rsHj. be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.55 SharePoint Server Database tzku Verbose ConnectionString: 'Data Source=HungerNet3;Initial Catalog=HNet3_Svc_Prod_UPA_Profile;Integrated Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Connect Timeout=15;Application Name=SharePoint[w3wp][3][HNet3_Svc_Prod_UPA_Profile]' Partition: NULL ConnectionState: Closed ConnectionTimeout: 15 be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.55 SharePoint Server Database ahjqp Verbose SQL connection time: 0.0456 for Data Source=HungerNet3;Initial Catalog=HNet3_Svc_Prod_UPA_Profile;Integrated Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Connect Timeout=15;Application Name=SharePoint[w3wp][3][HNet3_Svc_Prod_UPA_Profile] be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.55 SharePoint Portal Server User Profiles ajw8l High Exception in LoadRequestedUserProfile: (Microsoft.Office.Server.UserProfiles.UserNotFoundException: User Not Found: Could not load profile data from the database. at Microsoft.Office.Server.UserProfiles.UserProfile.Load(SqlDataReader myReader, Boolean bFirstRead, Boolean firstReaderIsViewerRights, Boolean includeColleagueRecords, Boolean includeLanguageAndRegionalSettings, Boolean shouldCloseReader) at Microsoft.Office.Server.UserProfiles.UserProfile.RetrieveUser(String strAcct, Guid gAcct, Byte[] bSid, Nullable`1 recordId, Boolean doNotResolveToMasterAccount, Boolean loadFullProfile, Boolean loadColleagueRecordIds, String strEmail) at Microsoft.Office.Server.UserProfiles.UserProfile..ctor(UserProfileManager objManager, String strAcct, Boolean doNotResolveToMasterAccount, Boole... be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.55* SharePoint Portal Server User Profiles ajw8l High ...an forceUserIsSelf, Boolean loadFullProfile, Boolean loadColleagueRecordIds, String strEmail) at Microsoft.Office.Server.UserProfiles.UserProfileManager.GetUserProfile(String strAccountName, Boolean doNotResolveToMasterAccount, Boolean loadFullProfile, Boolean loadColleagueRecordIds) at Microsoft.SharePoint.Portal.WebControls.ProfilePropertyLoader.LoadRequestedUserProfile(String& redirectionUrl)) be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.55 SharePoint Foundation General aat87 Monitorable be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.55 SharePoint Foundation General ajji6 High Unable to write SPDistributedCache call usage entry. be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.55 SharePoint Foundation Micro Trace uls4 Medium Micro Trace Tags: 0 nasq,20 ajji6,3 agb9s,10 ak8dj,1 b4ly,25 adyrv,1 adyrv,4 adyrv,3 ajw8l,1 aat87,3 ajji6 be11a89c-ebeb-a08d-57d2-a513d117e184
07/25/2014 14:11:08.55 SharePoint Foundation Monitoring b4ly Medium Leaving Monitored Scope (Request (GET:https://my.hungernet.org:443/Person.aspx?accountname=FA%255Ccbuchholz)). Execution Time=78.9958 be11a89c-ebeb-a08d-57d2-a513d117e184
Advertisement
Reply

Have Windows Updates been applied? See my note about this
https://twitter.com/imorrish/status/491020435039854592
Look at the URL in the search results. It may have invalid character in which case you can use this fix
http://blogs.msdn.com/b/spses/archive/2014/06/30/sharepoint-2013-the-search-results-with-5c-the-character-will-become-double-encoded-and-causes-broken-links.aspx
Regards,
Ian
http://sps.cloudapp.net
Regards, Ian Internet Sites running on SharePoint 2013 http://j.mp/sp2013sites