Quantcast
Viewing all 35736 articles
Browse latest View live

Enhancing accessibility in Project Online for better productivity

Today, professional teams and organizations across the world use Project Online, Microsoft’s project management platform, to streamline team processes, schedule tasks efficiently and manage resources and workflows for team projects. The platform stands out due to its highly-visual and intuitive interface.

However, many of the visual and design elements of Project Online have been inaccessible for users with visual impairment, deafness or motor disabilities. By incorporating accessibility considerations in the Project Online interface, we aim to address barriers for users with varying disabilities. Our team has been working on adding new and unique features to help users with disabilities leverage this online platform for better project outcomes.

Here’s how we made Project Online more accessible:

The challenge

Enhancing accessibility for a wide set of users is a key challenge considering the unique design and visual features of Project Online. Project Online is used by a wide range of users who rely on different features and tools to get the most out of this productivity tool. With the platform serving large teams that could be working and collaborating remotely, upgrades to the platform need to be measured and well-planned. Legacy screen elements need to be modified to work with nearly every screen reader and browser combination. Accessibility features need to consider varying degrees of disabilities and different user preferences. Features need to be customizable to help users with a diverse set of accessibility needs to make the most of their online productivity suite.

Screen reader enhancements

Users with visual impairments use a screen reader to detect and work with on-screen elements. These screen readers rely on Web Accessibility Initiative – Accessible Rich Internet Applications (WAI-ARIA) tags to help users identify and interact with elements.

Our efforts to enhance accessibility for Project Online were focused on ARIA tags. Although ARIA tags have been helpful for users with visual impairments, compatibility issues and legacy screen readers have led to users missing out on key elements and features of Project Online.  We rectified older ARIA tags and deployed software improvements to help screen readers work better with these tags. The result is that the latest version can differentiate between links, buttons, and other on-screen elements.

Image may be NSFW.
Clik here to view.

Screen reader highlighting the table outline for better navigability

While working with tables, the ARIA tags can help users navigate not only the whole table and select individual cells, but additionally read the titles and text inserted in each cell. The ability to highlight the whole table outline and edit specific cells improves navigability for users with visual impairments.

Upgraded ARIA tags also enable users to clearly identify images, graphs, and visual elements. Screen readers can now leverage dynamic swapper technology to announce a change or update to any cell in real-time and let users know if a mandatory field in a form hasn’t been completed. To address contrast issues for people with low or impaired vision, we changed the colors to adhere to 4.5:1 for text and background or foreground colors.  This solves the issue of users being unable to interact or identify information in tables, images, or forms due to the limitations of legacy screen reader software.

Image may be NSFW.
Clik here to view.

Color contrast between text and the background

These key enhancements for screen readers solve a wide range of usability issues for people with visual impairments. With upgrades and additions to keyboard shortcuts we have improved accessibility for users with physical disabilities, deafness, and low mobility.

Keyboard shortcuts for better productivity

Quick and easy keyboard shortcuts reduce the range of motion required to interact with the Project Online platform. Although a limited range of keyboard shortcuts have always been part of Project Online, the expansion of these shortcuts has enhanced the platform’s accessibility. A simple keystroke can now help users swap between URLs and the main content on a screen. This ensures users navigate the web page or move to a new web page when required without having to move a mouse or touch a screen.

Keyboard shortcuts help users swap between different elements on the screen. The Project Online dashboard is usually populated with intensive flowcharts, diagrams, tables, and graphics. By setting shortcuts, users with physical disabilities can easily navigate a whole table or data set to interact with specific elements on any page. They can apply the changes automatically across different pages of the same work portfolio. For users with visual impairments, changes were made so that the focus is on the foreground window on the screen. This enables the screen reader to announce the elements that pop up in the foreground, making it easier for users to navigate based on priority.

Keyboard shortcuts reduce the amount of effort required to interact with Project Online. With a few simple clicks users with disabilities are able to navigate the interface and reduce the time it takes to edit their project management workflows.

At Microsoft, we adopt a holistic design approach to address accessibility challenges. Having upgraded legacy elements for better compatibility with the latest screen readers, Project Online can now handle dynamic elements better.

Image may be NSFW.
Clik here to view.

The assistive technology interacts with browser controls and document object models (DOM) to augment the regular Project Online view. The features leverage the new and improved capabilities of screen readers and the ease-of-use of keyboard shortcuts to deliver better accessibility.

Better collaboration with better accessibility

From planning and scheduling tasks to managing resources and analyzing reports, project management encompasses a number of responsibilities. Highly intuitive and visual tools like Project Online have helped project managers and teams of all sizes collaborate and get work done efficiently.

Our work to enhance the functionality of screen readers, keyboard shortcuts, and screen contrast settings is aimed at improving accessibility of the project and team management platform further.

Enhancing accessibility is a key element of our mission to empower every person and every organization on the planet to achieve more. At Microsoft, our commitment to accessibility extends across our product spectrum as we endeavor to deliver equivalent experiences to people with disabilities. We’re committed to investing more and more efforts and resources in this direction to enable the shared goal of wider accessibility and inclusion.

For more information and support on accessibility features in Project Online, please visit https://support.office.com/en-us/article/accessibility-support-for-project-web-app-ff2ceabe-5f46-4b74-8444-4affe9b34fbb?ui=en-US&rs=en-US&ad=US

 


The PowerPC 600 series, part 12: leaf functions


On Windows NT for the PowerPC,
there is a leaf function optimization available provided
your function meets these criteria:




  • It calls no other functions.


  • It does not have an exception handler.


  • It does not need any stack space beyond
    stack space used by actual inbound parameters,
    the eight words of stack used as home space,¹
    and the 232-byte red zone.


  • It does not modify any nonvolatile registers.



If all of these conditions are met, then the function does
not need to declare any function unwind codes,
and it does not
need to set up a stack frame.
It can reuse the stack frame of its caller.
In order for the system to be able to unwind out of
a lightweight leaf function,
the leaf function must keep its return address in the lr
register throughout the entire life of the function,
and it cannot move the stack pointer.



Conversely, if you fail to declare unwind codes for a function,
then the system assumes that it is a lightweight leaf function.



Here's a sample function that is a candidate for lightweight leaf
status:



wchar_t* SkipLeadingSpacesAndTabs(wchar_t* s)
{
while (*s == L' ' || *s == L't') s++;
return s;
}


This is how the Microsoft compiler generated the code for it:



SkipLeadingSpacesAndTabs:
lhz r4,(r3) ; load wchar_t into r4
cmpwi cr6,r4,0x20 ; Is it a space?
beq cr6,loop ; Y: skip it
cmpwi cr7,r4,9 ; Is it a tab?
bne cr7,break ; N: done
loop:
lhzu r4,2(r3) ; Skip over current character and load next one
cmpwi cr6,r4,0x20 ; Is it a space?
beq cr6,loop ; Y: skip it
cmpwi cr7,r4,9 ; Is it a tab?
beq cr7,loop ; Y: continue
break:
blr ; Return to caller, result already in r3


For some reason, the Microsoft compiler likes to use
cr6 and cr7 as the targets for its comparison
instructions.
It probably wants to stay far away from cr0
and cr1,
which are implicitly updated by some instructions.



Notice that we used the lhzu instruction to
advance the r3 register and then fetch a halfword from it.
This shows how the update version of a load instruction is handy
for walking through an array.



If we wanted to be clever, we could apply the following transformation.
First, un-unroll the loop:



SkipLeadingSpacesAndTabs:
lhz r4,(r3) ; load wchar_t into r4
b test
loop:
lhzu r4,2(r3) ; Skip over current character and load next one
test:
cmpwi cr6,r4,0x20 ; Is it a space?
beq cr6,loop ; Y: skip it
cmpwi cr7,r4,9 ; Is it a tab?
beq cr7,loop ; Y: continue
break:
blr ; Return to caller, result already in r3


This seems like a pessimization, since we introduced a branch.
But now I can remove the branch by realizing that I can trick the
first iteration's lhzu to load the first halfword
of the string rather than the second:
Predecrement the value to counteract the preincrement!



SkipLeadingSpacesAndTabs:
subi r3,r3,2 ; decrement to counteract the upcoming increment
loop:
lhzu r4,2(r3) ; Skip over current character and load next one
cmpwi cr6,r4,0x20 ; Is it a space?
beq cr6,loop ; Y: skip it
cmpwi cr7,r4,9 ; Is it a tab?
beq cr7,loop ; Y: continue
break:
blr ; Return to caller, result already in r3


Finally, I can combine the results of the two comparisons so there
is only one branch that needs to be predicted:


SkipLeadingSpacesAndTabs:
subi r3,r3,2 ; decrement to counteract the upcoming increment
loop:
lhzu r4,2(r3) ; Skip over current character and load next one
cmpwi cr6,r4,0x20 ; Is it a space?
cmpwi cr7,r4,9 ; Is it a tab?
cror 4*cr7+eq,4*cr6+eq,4*cr7+eq ; Is it either?
beq cr7,loop ; Y: continue
blr ; Return to caller, result already in r3


I don't know whether this performs better than the original code,
but it
is four instructions shorter,
consumes one fewer branch prediction slot,
and simply looks cooler.
I win on style points,
but I could very well lose on real-world performance.



¹
As I noted earlier, you are allowed to use all of the home space
even if your function doesn't have that many parameters.

.NET Core August 2018 Update

Today, we are releasing the .NET Core August 2018 Update. This update includes .NET Core 2.1.3 and .NET Core SDK 2.1.401.

Getting the Update

The latest .NET Core updates are available on the .NET Core download page.

See the .NET Core 2.1.3 release notes for details on the release including a detailed commit list.

Docker Images

.NET Docker images have been updated for today’s release. The following repos have been updated.

microsoft/dotnet
microsoft/dotnet-samples

Note: Look at the “Tags” view in each repository to see the updated Docker image tags.

Note: You must re-pull base images in order to get updates. The Docker client does not pull updates automatically.

2.1 declared LTS

As mentioned in a previous blog post, we have been working toward declaring .NET Core 2.1 as LTS. With the August Update, we're pleased to announce that 2.1.3 begins the .NET Core 2.1 LTS lifecycle. We'll continue to update 2.1 with important fixes to address security and reliability issues as wells as add support for new operating system versions. Details on the .NET Core lifecycle can be seen in the .NET Core support policy.

What's covered?

The LTS designation covers all packages referenced by Microsoft.NETCore.App and Microsoft.AspNetCore.App. To ensure that applications running on 2.1 remain on the LTS supported updates, it's important to reference these 'meta-packages' rather than individual 'loose' packages. This will enable the applications to properly utilize 2.1 updates when they are released and installed.

The .NET Core SDK is not included as part of the LTS designation. SDK fixes will continue to be released in the latest version 'train' which supports maintaining and building applications for .NET Core 2.1.

See .NET Core Supported OS Lifecycle Policy to learn about Windows, macOS and Linux versions that are supported for each .NET Core release.

Previous .NET Core Updates

The last few .NET Core updates follow:

July 2018 Update
June 2018 Update
May 2018 Update

Modern Security- & Load-Testing with VSTS when bound to on-premises

In many enterprises Azure is the strategic public cloud. But it still can have technical or organizational blockers which prevent you from fully profiting from the public cloud. Embracing modern DevOps practices to optimize and simplify the process of creating end user value is much harder if you build on top of legacy processes and hardware. Based on a real life scenario I'm describing in this post how you can use cloud technology to practice at least a bit of DevOps even if your organization is still not ready yet for public cloud.

The scenario

We are talking about a massive eCommerce web application (built with ASP.Net Core) that faces the extra challenge of not being able to leverage public cloud technologies. The development team for this eCommerce platform consists of 30 developers which contribute from two locations (onsite & nearshore). They are sharing an on-prem TFS 2018 with other dev-teams but have their dedicated build server. The application itself runs on Windows behind an IIS webserver.

Beside using latest ASP.Net Core technology this is pretty much the classical way how application are getting developed and deployed. The application is ready to be containerized, but external dependencies do not allow a deployment to a Docker-hosted environment at the moment.

How VSTS & Azure still enable DevOps

As the platform will have a massive user load it is absolutely crucial to test the application under load on a regular base starting already early in the development process. But the initiative of getting a dedicated on-prem environment with (1st) the load-generator agents and (2nd) the application itself to run the load against would probably have died due to complexity on an early stage.

To unlock this situation we have chosen a fully cloud based solution. As a prerequisite all relevant interfaces of the application have been mocked (https://en.wikipedia.org/wiki/Mock_object). Then the application gets deployed to an Azure VM (automated though TFS Release Management) and the cloud-based load-test from VSTS simulate user-load for the application running on the Azure VM. In addition to this Application Insights (https://azure.microsoft.com/en-us/services/application-insights/) is activated only on the application instance running on this Azure VM what provides us dozens of additional metrics while the application is under load.

Image may be NSFW.
Clik here to view.

Continuously more and more important is security testing and especially considering the OWASP (https://www.owasp.org) threats. Executing scans with the ZAP (Zed Attack Proxy, https://github.com/zaproxy/zaproxy/wiki/Downloads) again requires hardware for this proxy-software as well as for triggering the requests.

Extending the scenario from above with a second Azure VM which hosts the proxy and a VSTS build definition which makes use of the test task coming with this VSTS extension (https://marketplace.visualstudio.com/items?itemName=kasunkodagoda.owasp-zap-scan) is our easy and efficient solution to run OWASP security tests on a regular base.

Image may be NSFW.
Clik here to view.

What if…

With this solution cloud based security- and load-tests are possible although no data (not even source code) is stored in the public cloud. But without any boundaries in leveraging public cloud (VSTS & Azure) the ALM (Application Lifecycle Management) process could still be much more efficient:

  1. Containerizing the application and running it as micro-services on a managed infrastructure (e.g. AKS, https://docs.microsoft.com/en-us/azure/aks/) would give you full flexibility in terms of scalability and provisioning additional environments by reducing the operational effort to rare minimum.
  2. The execution of the mentioned Security- & Load-Tests could be orchestrated as part of the existing build-, test- and release-pipeline
  3. When working on VSTS you automatically profit from the newest features and get rid of upgrading TFS once or twice a year.

The SQL Server Defensive Dozen – Part 3: Authentication and Authorization in SQL Server

Introduction

In order to secure and harden SQL Server it is important to control who and/or what can authenticate to the system and what they have access to. This will satisfy regulatory needs such as the Secure Technical Implementation Guide (STIG).

As a reminder from our Introduction to this series, hardening is the process of applying Administrative and Technical controls to a system. Every article in this series will review both Administrative Controls and Technical Controls needed to properly managed and harden a system. This article will discuss authentication configuration and documentation for hardening SQL Server.

Administrative Controls

Documentation is essential to the success of a secured SQL Server. Ideally, documentation should be created before SQL Server is even installed. This allows the DBA to build SQL Server tailored to the requirements of the project and should allow SQL Server to operate in a more secure state from start to finish. However, it is not always realistic to build the documentation prior to installing SQL Server. In such cases it is recommended to ensure the project is configured to a working state, then back down the permissions and turn off any feature which are not required. Once it is determined what the minimal level of permissions and features are required for operation, these can then be documented.

Authorizing access to principals within SQL Server is vital to the success of maintaining a secure environment. Without thorough documentation, the DBA will have the difficult (if not impossible) task of keeping SQL Server secure. Thorough documentation ensures the DBA has written guidance on how the data is to be kept secure and provides accountability to the organization to ensure a least privilege model is maintained.

Many hardening guides require the use of Windows Authentication (Microsoft's recommendation) coupled with password complexity requirements (typically inherited from Group Policy) for compliance. In the event that Mixed Mode Authentication is utilized, the information owners must choose whether to inherit the password complexity requirements from the Operating System (OS), or if they will implement password complexity requirements utilizing policies and procedures defined by the organization. All of these decisions must be documented.

Additionally, documenting the authorized principals and permissions within SQL Server and regularly comparing them to a properly vetted baseline is essential to ensure unauthorized access is not granted. This can be accomplished using a tool like the SQL Server Vulnerability Assessment.

Technical Controls

The technical controls discussed in this section implement and enforce the administrative controls which were defined in the Administrative Controls section. These controls are settings which can be configured within SQL Server to ensure the written policy is enforced.

Authentication Mode

Configuring the authentication mode in accordance with system documentation is vital to the integrity and trustworthiness of the system. The following sections discuss various technical procedures which should be followed when implementing the Windows or Mixed authentication modes.

All Authentication Modes

  • Rename and disable the SA login to prevent attacks on the well-known login by account name.  Automated attacks will try to brute force the SA login as soon as a SQL instance is discovered. By renaming the SA login, most automated attacks will fail to brute force the database admin password.
  • Change SA password to a strong password. This is valuable even in Windows Authentication Mode in case Mixed Authentication Mode is enabled inadvertently or intentionally. Store the password in a secure location for a "break glass in case of emergency" situation.
  • Do not delete built-in principals. This will result in the loss of functionality and potentially an inaccessible system.
  • Use logon triggers for more granular control of the login process.
  • Using either local or Active Directory policies, implement a password policy including expiration and complexity requirements at a minimum.
  • Ensure passwords are encrypted when stored in client applications.

Windows Authentication

  • Windows authentication is the preferred and recommended form of authentication.
  • Use Windows groups to authorize users rather than Windows logins where possible to reduce the administrative overhead of managing access to the instance. It is important to note that each Windows user will be individually identified in the audit log, even if they are a member of a Window group. For example, if the domain user ContosoDan is a member of the domain group ContosoDBA, the user ContosoDan will be identified in the SQL Server audit logs.

Mixed Mode Authentication

  • Use for users from untrusted domains/workgroups, cross-platform users, non-Windows applications, or legacy applications.
  • Do not use the SA login to manage SQL Server. Rather assign a known principal to the appropriate roles (e.g. sysadmin).
  • Outfit applications with a mechanism to change SQL login passwords.
  • Set MUST_CHANGE for new logins to ensure the end user changes the password to something only they know.

Baseline Logins and Users Principals and Permissions

Principals represent users, groups, services, and other entities which can access SQL Server resources. Principals gain permissions through direct assignment or role membership.

Verifying and limiting the use of permissions and roles inside of SQL Server is a requirement of many hardening guides. Administrators are encouraged to baseline the documented role memberships and server-level permissions. Administrators are also advised to regularly review (audit) the baseline against production to verify nothing has changed. The SQL Server Vulnerability Assessment tool can be used to help with these actions.

In most cases, the hardening guides mandate the permission or role membership must follow the Principle of Least Privilege. Best practices for implementing the principle of least privilege are:

The following are queries to extract SQL Server permission and role information. These queries are useful for creating and validating baselines, creating and validating documentation, and investigating permissions granted to principals.

Login List

Obtain all logins of the following types:

  • Certificate Login
  • SQL Login
  • Windows Group
  • Windows Login
SELECT  login.name,
        login.type_desc,
        CASE login.is_disabled
            WHEN 1 THEN 'disabled' 
            ELSE 'enabled'
        END AS 'Status'
FROM    sys.server_principals login
WHERE   login.type IN ('C','G','S','U')
ORDER   BY login.name

Direct Permissions Assignment to Logins

Obtain all the permissions for the logins of the following types:

  • Certificate Login
  • SQL Login
  • Windows Group
  • Windows Login
SELECT  login.name,
        login.type_desc,
        perm.state_desc,
        perm.permission_name,
        perm.class_desc
FROM    sys.server_principals login 
JOIN    sys.server_permissions perm ON login.principal_id = perm.grantee_principal_id 
WHERE   login.type IN ('C','G','S','U')
ORDER   BY login.name

Server Role Permissions Assignment

Obtain all the permissions for the server roles.

SELECT  role.name,
        CASE role.is_fixed_role
            WHEN 0 THEN 'False'
            ELSE 'True'
        END AS 'Built In Role',
        perm.state_desc,
        perm.permission_name,
        perm.class_desc
FROM    sys.server_principals role
JOIN    sys.server_permissions perm ON role.principal_id = perm.grantee_principal_id
WHERE   role.type = 'R'
ORDER   BY role.name

Server Role Membership

Obtain the server principals which are members of the server roles.

SELECT  role.name AS 'Role',
        login.name AS 'Login'
FROM    sys.server_principals role 
JOIN    sys.server_role_members rm ON role.principal_id = rm.role_principal_id 
JOIN    sys.server_principals login ON rm.member_principal_id = login.principal_id
ORDER   BY role.name

Shared Accounts

No accounts should access the system that cannot be individually identified and audited. Non-repudiation of actions taken is required to maintain system confidentiality, integrity, and availability. Non-repudiation protects against later claims by a user of not having created, modified, or deleted objects or data in the database or instance.

NT AUTHORITYSYSTEM

Ensure the NT AUTHORITYSYSTEM (SYSTEM) account is not privileged or otherwise documented as having elevated access. Rather than granting SYSTEM additional permissions for services running as SYSTEM, consider using a Service SID to grant permissions directly to the service itself. This is the method SQL Server uses to grant permissions to the SQL Server and SQL Agent services. This method has been adopted by Microsoft System Center Operations Manager (SCOM) administrators to grant permission to the HealthService within SQL server.

Any user with enough access to the server can execute a task that will be run as SYSTEM either using task scheduler or other tools. At this point, SYSTEM essentially becomes a shared account because the operating system and SQL Server are unable to determine who created the process. This defeats non-repudiation as the actions taken by SYSTEM show in the audit logs as having been taken by SYSTEM, and not the user who invoked the command.

Prior to SQL Server 2012, NT AUTHORITYSYSTEM was a member of the sysadmin role by default. In more recent versions of SQL Server, NT AUTHORITYSYSTEM is explicitly not given sysadmin access.

Database Ownership

Databases should be owned by a login which should have full control of the database. The database owner login is mapped to the dbo user in the database. The dbo user has full control in the database and cannot be limited. It is recommended to use a low-privilege login as the owner of the databases. A low-privilege login should have no administrative access to the instance. This is especially important in cases where TRUSTWORTHY is enabled on the database, because the use of TRUSTWORTHY combined with a high-privilege login can be used to escalate the privileges of a low-privilege user within the database.

Use the ALTER AUTHORIZATION statement to change the owner of the database.

Contained Databases

SQL Server 2012 introduced the concept of contained databases. A contained database is a database which includes all the settings and metadata needed for its operation without dependencies on the instance. From a security perspective, a contained database makes it easier to have user accounts that are limited to a single database. Contained databases should only be enabled if they are required. As with the instance, Windows users or groups should be used when creating users within the contained database.

Discretionary Access Control (GRANT WITH GRANT)

Discretionary access control is based on the notion that principals are "owners" of objects and therefore have discretion over who should be authorized to access the object and in which mode (e.g., read or write). Ownership is implicitly or explicitly defined during object creation. Discretionary access control allows the owner to determine who will have access to objects they control.

The follow queries can be used to determine who has been granted GRANT with GRANT:

Server Level

SELECT  who.name AS [Principal Name],
        who.type_desc AS [Principal Type],
        who.is_disabled AS [Principal Is Disabled],
        what.state_desc AS [Permission State],
        what.permission_name AS [Permission Name]
FROM    sys.server_permissions what
INNER   JOIN sys.server_principals who ON who.principal_id = what.grantee_principal_id
WHERE   what.state_desc= 'GRANT_WITH_GRANT_OPTION'

Database Level

USE [DatabaseName]
GO

SELECT  who.name AS [Principal Name],
        who.type_desc AS [Principal Type],
        what.state_desc AS [Permission State],
        what.permission_name AS [Permission Name]
FROM    sys.database_permissions what
INNER   JOIN sys.database_principals who ON who.principal_id = what.grantee_principal_id
WHERE   what.state_desc= 'GRANT_WITH_GRANT_OPTION'

Proxies and External Access

Customers may have designated administration user and/or service accounts on the operating system. Typically, this is an administration account with which they automate complex maintenance routines (such as automatically adding a database to an Availability Group).

SQL Server provides highly-secure features (also capable of being permissioned) to perform tasks. Please see How to: Create a Credential (SQL Server Management Studio) and Creating SQL Server Agent Proxies for more information.

Remove any SQL Agent Proxy accounts and credentials that are not authorized.

Conclusion

This completes the facet and considerations related to securing SQL Server authentication and authorization. Please stay tuned for the next topic in the series discussing modules management and programmability for SQL Server.

What’s new in VSTS Sprint 138 Update

The Sprint 138 Update of Visual Studio Team Services (VSTS) has rolled out to all organizations. In this update, test results for releases have been enhanced to group together test attempts and summarize better. Watch the following video to learn about this and a few of the other new features.

Now you can also bring your dashboard to Microsoft Teams using the Dashboard tab available in the Microsoft Teams Integration extension. Romi describes this more in her blog post.

Image may be NSFW.
Clik here to view.
Dashboard embeeded in MS Teams

Check out the release notes for more details and consider leaving a comment below if you’re looking for clarification on any of the new features in this Update.

A Microsoft DevSecOps Static Application Security Testing (SAST) Exercise

Static Application Security Testing (SAST) is a critical DevSecOps practice. As engineering organizations accelerate continuous delivery to impressive levels, it’s important to ensure that continuous security validation keeps up. To do so most effectively requires a multi-dimensional application of static analysis tools. The more customizable the tool, the better you can shape it to your actual security risk.

Static Analysis That Stays Actionable and Within Budget: A Formula

Let’s agree that we employ static analysis in good faith to improve security (and not, for example, to satisfy a checklist of some kind). With that grounding, static analysis is like a tripod: it needs three legs to stand:

  1. Relevant patterns.
  2. Cost-effective mechanisms to detect and assess.
  3. Business motivation to fix.

Image may be NSFW.
Clik here to view.
Improvement depends on all three legs: relevant patterns, motivation to fix, and cost-effective detection

There’s little point engaging if any element is missing - you can’t argue a two-legged stool into working as it should.

Here’s a formula that captures a healthy static analysis process. Has this alert ever gone red for you? The red mostly materializes in faces. The (hardly lukewarm) language that spews from mouths is blue.

Image may be NSFW.
Clik here to view.
The number of issues times (cost to produce plus cost to evaluate plus cost to resolve) divided by the product of impact times exposure times true positive rate is less than or equal to the motivation to address

This formula expresses the experience that value delivered is only in fixes for discovered issues. Everything else is waste. If we can correlate (or restrict) findings to those that lead to actual product failures, for example, we should see greater uptake.

Too many results overwhelm. Success is more likely with fewer false positives (reports that don’t flag a real problem). We can increase return on investment by looking for improvements in each formula variable. If the cost to resolve a result is high, that may argue for improving tool output to make it clearer and more actionable.

I’ve encoded security budget as ‘motivation to address’ (and not in terms of dollars or developer time) to make the point that a team’s capacity expands and contracts in proportion to the perceived value of results. A false positive that stalls a release is like a restaurant meal that makes you sick: it is much discussed and rarely forgotten. The indigestion colors a team’s motivation to investigate the next problem report.

A report that clearly prevents a serious security problem is a different story entirely and increases appetite. To succeed, therefore, analysis should embody the DevSecOps principle that security effort be grounded in actual and not speculative risk.

Multiple Security Test Objectives: Unblock the Fast and Thorough

“I know your works; that you are neither cold nor hot. Would that you were cold or hot! But because you are lukewarm, and neither cold nor hot, I will spew you from my mouth.” – Revelations 3:15 – 16.

A core problem with realizing the DevSecOps vision is that our developer inner loop doesn’t provide relevant security data. It is designed, rather, for the highly accelerated addition of new functionality (which inevitably adds security risk). Tools-driven code review can help close the gap.

To ensure that continuous security validation is pulling its weight without slowing things down, I recommend to teams at Microsoft that they divide results into three groups:

  1. The ‘hot’: lightweight, low noise results that flag security problems of real concern, with low costs to understand and investigate. These results are pushed as early into the coding process as possible, typically blocking pull requests. Example find: a hard-coded credential in a source file.
  2. The ‘cold’: deep analysis that detects serious security vulnerabilities. These results are produced at a slower cadence. They cost more to investigate and fix and are undertaken as a distinct activity. Results are filed as issues which are prioritized, scheduled and resolved as usual. A high-severity warning might block a release, but it should not block development. Example find: a potential SQL injection issue with a complex data-flow from untrusted source to dangerous sink.
  3. The ‘lukewarm’: everything else. Spew these low values results from your process mouth and recover the budget. Use the freed cycles to fund a ‘cold’ path analysis (if it doesn’t exist) or spend it on more feature work (if it does).

Besides identifying security vulnerabilities that are actually exploitable in code (we want these!), the cold path yields insights for tuning hot path analysis to keep it focused on what counts. And if you use the proper tools, this practice can be used to bring high impact analysis back to the inner loop, by authoring new checks that codify the reviewer’s knowledge of security and your project’s attack surface area.

Image may be NSFW.
Clik here to view.
The cold path workflow

A successful cold path workflow first finds and fixes the original bug, then finds and fixes variants, and then monitors to prevent recurrence. Let’s look a concrete example of how this works at Microsoft.

An MSRC Secure Code Review Case Study

The Microsoft Security Response Center (MSRC) recently described a security review to me that integrated static analysis as a central component for the cold path (#2 above). It was striking how things lined up to make this deployment of static analysis an unqualified success.

The analysis engine utilized, authored by Semmle, converts application code into a database that can be queried via a scripting language (‘Semmle QL’). The query engine is multi-language and can track complex data flows that cross function or binary boundaries. As a result, the technology is quite effective at delivering high value security results at low costs relative to other analysis platforms. Today, MSRC and several other teams at Microsoft have 18+ months of evaluations and rules authoring, with thousands of bugs filed and fixed as proof points for the tech.

You can see read more about MSRC’s Semmle use in great technical detail on their blog. Here’s the setup for the effort we’re discussing here:

  • In an end-to-end review of a data center environment, MSRC flagged some low-level control software as a potential source of problems. The risk related both to code function and its apparent quality.
  • Because this source code was authored externally, Microsoft had no engineers with knowledge of it.
  • Multiple versions of the code had been deployed.
  • The code base was too large for exhaustive manual review.
  • Due to the specialized nature of the software, no relevant out-of-box analysis existed for it. [This is typical. The more serious the security issue you’re looking for, the greater the likelihood you will need a custom or highly tuned analysis for acceptable precision and recall. Semmle shines in these cases due to the expressiveness of its authoring language and the low costs to implement and refine rules.]

This code base was deemed sufficiently sensitive to warrant MSRC review. If you’ve worked with a penetration-test or other advanced security review team, you know one way this plays out. The reviewer disappears for days or weeks (or months), emerges with several (or many) exploitable defects, files bugs, authors a report and moves on, often returning to help validate the product fixes that roll in. This exercise, supported by Semmle QL, showed how things can go when ‘powered up’ by static analysis:

  • First, a security engineer compiled and explored the code to get familiar with it.
  • After getting oriented, the engineer conducted a manual code review, identifying a set of critical and important issues.
  • The reviewer prioritized these by exposure and impact and a second engineer authored Semmle QL rules to locate the top patterns of interest.
  • Next, the rules author ran the analysis repeatedly over the code base to improve the analysis. The manually identified results helped eliminate ‘false negatives’ (that is, the first order of business was to ensure that the tool could locate every vulnerability that the human did). New results (found only by the tool) were investigated and the rules tuned to eliminate any false positives.

Two engineers worked together on this for three days. The productive outcomes:

  • Static analysis tripled the number of total security findings (adding twice as many new results to those located by manual review).
  • The engineers were able to apply the static analysis to multiple variants of the code base, avoiding a significant amount of (tedious and error prone) effort validating whether each defect was present in every version. This automated analysis in this way also produced a handful of issues unique to a subset of versions (contributing to the 3x multiplier on total findings).
  • As fixes were made, the analysis was used to test the product changes (by verifying that bugs could no longer be detected in the patched code).
  • On exit, the security team was left with an automatable analysis to apply when servicing or upgrading to new versions of the code base.

Continuously Reexamining Continuous Security Validation

Deep security work takes time, focus, and expertise that extends beyond a mental model of the code and its operational environment (these are essential, too, of course). Static analysis can clearly accelerate this process and, critically, capture an expert’s knowledge of a code base and its actual vulnerabilities in a way that can be brought back to accelerate the development inner loop.

Semmle continues to partner in the static analysis ecosystem for Microsoft, the industry at large and the open source community. Microsoft and Semmle are participants on a technical committee, for example, that is in the final stages of defining a public standard for persisting static analysis results (‘SARIF’, the Static Analysis Results Interchange Format). The format is a foundational component for future work to aggregate and analyze static analysis data at scale. We are working together to release a body of Semmle security checks developed internally at Microsoft to the open source community. We continue partnering to protect many of Microsoft’s security-critical code bases.

Analytics Private Preview for Customers on TFS 2018 Update 2

Analytics is the new reporting platform for both Team Foundation Server (TFS) and Visual Studio Team Services (VSTS).

We are starting a Limited Private Preview of Analytics for TFS 2018 Update 2 in preparation for a full release in TFS 2019.

To learn more about Analytics:

We are looking for TFS 2018.2 customers who would be willing to join the preview.

The Preview has full support for TFS 2019 upgrades and additional direct support for Analytics (including fixes for blocking issues) for those customers involved. 

We are specifically looking for customers who are willing to help us by running Analytics on their production TFS 2018 instances and providing us with key metrics on its performance.

Please consider the following:

  • We would strongly prefer customers that are willing to deploy in a production environment which is being actively used for work item tracking.
  • We will ask customers to periodically monitor SQL perf so we can better understand Analytics resource consumption
  • If a customer finds Analytics unacceptable for any reason, we will provide a path to turn Analytics off and remove any Analytics data.
  • If a customer finds any blocking issues we will provide a bug fix directly.
  • This private preview includes Analytics-based widgets and OData access.
  • It does not include Power BI + Analytics Views integration using our Power BI Data Connector

The Preview is limited. We may not be able to accept everyone wanting to be a part of the preview.

If you are interested please reach out to us at tfs2018axpreview@microsoft.com.

Thank you

 


Azure Web App for Containers をご利用のお客様へのお知らせ (Docker Hub のメンテナンスについて)

Azure Web App for Containers のカスタム Docker イメージ ソースとして Docker Hub をご利用のお客様につきまして、サービス稼働に影響のある情報がありますので、お知らせします。

 


先日 Docker 社より、2018 年 8 月 25 日 11:00 AM (PST) から、Docker 各サービスについて 15 分から 45 分のアップグレード作業が実施されることがアナウンスされました。以下、アナウンス情報をご確認ください。

Planned Downtime on Docker Hub, Docker Cloud, and Docker Store on August 25th, 2018

 

このアップグレード作業の実施中には、Docker Hub について、Docker イメージの Push および Docker イメージの Pull ができなくなるとのことです。したがいまして、日本時間の 8 月 26 日 (日) 4:00 AM から、一時的に Docker Hub からの Docker イメージの Pull ができない状態になる可能性があります。

Azure Web App for Containers では、アプリケーションをホストするインスタンスの初期化処理において、Docker イメージを Pull し、Docker コンテナを起動します。このため、Docker イメージを Pull できない場合、インスタンスの初期化に失敗します。
そして、インスタンスの初期化に失敗したインスタンスに対して要求が転送されることで、HTTP エラーが応答される動作となる場合があります。

なお、本件は Web App for Containers でカスタム Docker イメージを利用している場合が対象となります。App Service on Linux で組み込みのランタイム スタックを利用している場合には、Docker Hub へのアクセスは発生しませんので、本件による影響はありません。

 

 

以上の理由により、Azure Web App for Containers のカスタム Docker イメージ ソースとして Docker Hub を設定している場合、Docker Hub 側でのアップグレード作業実施中に、Web App 側で以下のような動作が発生すると、サービス稼働に影響が及ぶ可能性があります。

  • Web App の再起動

Web App の再起動により、Docker イメージの再展開が行われるため、影響を受ける場合があります。これには、手動による再起動の他、Azure プラットフォーム側でサービス品質の低下を検知したことによる自動再起動も含まれます。
この場合、再起動が行われたインスタンスが影響を受けることになります。

  • インスタンスのスケール アップ

インスタンス サイズを変更することにより、新規サイズのインスタンスへの Docker イメージの展開が行われるため、影響を受ける場合があります。
この場合、全インスタンスが影響を受けることになります。

  • インスタンスのスケール アウト

インスタンス数を増加させることにより、新規インスタンスへの Docker イメージの展開が行われるため、影響を受ける場合があります。
この場合、増加分のインスタンスが影響を受けることになります。

 

 

現時点では、Azure Web App for Containers では、Docker Hub にアクセスできないという状況についての対策方法がございません。
このため、上記のような問題を回避するために、現状では、Azure Container Registry あるいはその他のプライベート レジストリを利用するということが対策案となります。
該当の時間帯に影響を受ける可能性のあるアプリケーションをホストしている場合、前述のような動作が発生した場合にサービス稼働に影響が及ぶ可能性がありますので、何卒対応をご検討ください。

Protected: Build Intelligent Web App with Machine Learning Service

This content is password protected. To view it please enter your password below:

Public Environment Repository in Labs

We are happy to announce that we are releasing a Public Repository of ARM templates for each lab that you can use to to create environments, without having to connect to an external GitHub source by yourself. The repository includes frequently used templates such as Azure Web Apps, Service Fabric Cluster and Dev SharePoint Farm environment. This feature is similar to the public repository of artifacts that is included today for every lab that you create. The environment repository will allow you to quickly get started with pre-authored environment templates with minimum input parameters to provide you with a smooth getting started experience for PaaS resources within labs. Note that we will keep adding more templates to this repository to simplify environment creation. We would also like to inform you that this will be an open source repository that you can contribute into, to add frequently used and helpful ARM templates of your own. To contribute, simply submit a pull request to the repository at the DevTest Labs GitHub Repo and we will review it.

Configuring Public Environments

As a lab owner, you will be able to enable the public environment repository for your lab during lab creation. Upon selecting 'On', the repo will be enabled for your lab. Note that, for existing labs the public environment repository will be disabled. You will have to explicitly enable it to leverage the templates within. For labs created using ARM templates, the repository will be disabled by default as well.

Image may be NSFW.
Clik here to view.

You can manage access to these environments via the newly added Public Environments blade in labs. Not only will you be able to enable/disable these environments but also white-list the one's that interest you/your lab users the most. To access the blade, select ‘Configuration and policies’ option; under the ‘Virtual machine bases’ subsection select the ‘Public environments’ which will open the following blade.

Image may be NSFW.
Clik here to view.

Here, you will be able to enable the Public Environments repository using the Yes/No toggle. When the repository is enabled, you can white list individual environment templates by checking or un-checking them. This will create a policy that will only allow the checked environments to be available to your lab users in the environment creation flow.

Using Environment Templates

As a lab user, you can create a new environment from the enabled list of environment templates by simply selecting +Add from the lab tool bar. The list of bases will include the public environments templates enabled by your lab admin at the top of the list.

Image may be NSFW.
Clik here to view.

The list of public environments include:

  • Service Fabric Lab Cluster
  • SharePoint 2013 Development Farm
  • Web App
  • Web App with MySQL
  • Web App with PostgreSQL
  • Web App with SQL Database

We hope you find this feature useful!

Try it today and let us know what you think about it!

Got an idea to make it work better for you? Submit your feedback/ideas or vote for others at Azure DevTest Labs User Voice forum.

Have a question? Check out the answers or ask a new one at MSDN forum.

std::string_view: The Duct Tape of String Types

Visual Studio 2017 contains support for std::string_view, a type added in C++17 to serve some of the roles previously served by const char * and const std::string& parameters. string_view is neither a "better const std::string&", nor "better const char *"; it is neither a superset or subset of either. std::string_view is intended to be a kind of universal "glue" -- a type describing the minimum common interface necessary to read string data. It doesn't require that the data be null-terminated, and doesn't place any restrictions on the data's lifetime. This gives you type erasure for "free", as a function accepting a string_view can be made to work with any string-like type, without making the function into a template, or constraining the interface of that function to a particular subset of string types.

tl;dr

string_view solves the "every platform and library has its own string type" problem for parameters. It can bind to any sequence of characters, so you can just write your function as accepting a string view:

void f(wstring_view); // string_view that uses wchar_t's

and call it without caring what stringlike type the calling code is using (and for (char*, length) argument pairs just add {} around them)

// pass a std::wstring:
std::wstring& s;         f(s);

// pass a C-style null-terminated string (string_view is not null-terminated):
wchar_t* ns = "";        f(ns);

// pass a C-style character array of len characters (excluding null terminator):
wchar_t* cs, size_t len; f({cs,len});

// pass a WinRT string
winrt::hstring hs;       f(hs);

f is just an ordinary function, it doesn’t have to be a template.

string_view as a Generic String Parameter

Today, the most common "lowest common denominator" used to pass string data around is the null-terminated string (or as the standard calls it, the Null-Terminated Character Type Sequence). This has been with us since long before C++, and provides clean "flat C" interoperability. However, char* and its support library are associated with exploitable code, because length information is an in-band property of the data and susceptible to tampering. Moreover, the null used to delimit the length prohibits embedded nulls and causes one of the most common string operations, asking for the length, to be linear in the length of the string.

Sometimes const std::string& can be used to pass string data and erase the source, because it accepts std::string objects, const char * pointers, and string literals like "meow". Unfortunately, const std::string& creates "impedance mismatches" when interacting with code that uses other string types. If you want to talk to COM, you need to use BSTR. If you want to talk to WinRT, you need HSTRING. For NT, UNICODE_STRING, and so on. Each programming domain makes up their own new string type, lifetime semantics, and interface, but a lot of text processing code out there doesn't care about that. Allocating entire copies of the data to process just to make differing string types happy is suboptimal for performance and reliability.

Example: A Function Accepting std::wstring and winrt::hstring

Consider the following program. It has a library function compiled in a separate .cpp, which doesn't handle all string types explicitly but still works with any string type.

// library.cpp
#include <stddef.h>
#include <string_view>
#include <algorithm>

size_t count_letter_Rs(std::wstring_view sv) noexcept {
    return std::count(sv.begin(), sv.end(), L'R');
}
// program.cpp
// compile with: cl /std:c++17 /EHsc /W4 /WX
//    /I"%WindowsSdkDir%Include%UCRTVersion%cppwinrt" .program.cpp .library.cpp
#include <stddef.h>
#include <string.h>
#include <iostream>
#include <stdexcept>
#include <string>
#include <string_view>

#pragma comment(lib, "windowsapp")
#include <winrt/base.h>

// Library function, the .cpp caller doesn't need to know the implementation
size_t count_letter_Rs(std::wstring_view) noexcept;

int main() {
    std::wstring exampleWString(L"Hello wstring world!");
    exampleWString.push_back(L'\0');
    exampleWString.append(L"ARRRR embedded nulls");
    winrt::hstring exampleHString(L"Hello HSTRING world!");

    // Performance and reliability is improved vs. passing std::wstring, as
    // the following conversions don't allocate and can't fail:
    static_assert(noexcept(std::wstring_view{exampleWString}));
    static_assert(noexcept(std::wstring_view{exampleHString}));

    std::wcout << L"Rs in " << exampleWString
        << L": " << count_letter_Rs(exampleWString) << L"n";

    // note HStringWrapper->wstring_view implicit conversion when calling
    // count_letter_Rs
    std::wcout << L"Rs in " << std::wstring_view{exampleHString}
        << L": " << count_letter_Rs(exampleHString) << L"n";
}

Output:

>.program.exe
Rs in Hello wstring world! ARRRR embedded nulls: 4
Rs in Hello HSTRING world!: 1

The preceding example demonstrates a number of desirable properties of string_view (or wstring_view in this case):

vs. making count_letter_Rs some kind of template
Compile time and code size is reduced because only one instance of count_letter_Rs need be compiled. The interface of the string types in use need not be uniform, allowing types like winrt::hstring, MFC CString, or QString to work as long as a suitable conversion function is added to the string type.
vs. const char *
By accepting string_view, count_letter_Rs need not do a strlen or wcslen on the input. Embedded nulls work without problems, and there's no chance of in-band null manipulation errors introducing bugs.
vs. const std::string&
As described in the comment above, string_view avoids a separate allocation and potential failure mode, because it passes a pointer to the string's data, rather than making an entire owned copy of that data.
string_view For Parsers

Another place where non-allocating non-owning string pieces exposed as string_view can be useful is in parsing applications. For example, the C++17 std::filesystem::path implementation that comes with Visual C++ uses std::wstring_view internally when parsing and decomposing paths. The resulting string_views can be returned directly from functions like std::filesystem::path::filename(), but functions like std::filesystem::path::has_filename() which don't actually need to make copies are natural to write.

inline wstring_view parse_filename(const wstring_view text)
	{	// attempt to parse text as a path and return the filename if it exists; otherwise,
		// an empty view
	const auto first = text.data();
	const auto last = first + text.size();
	const auto filename = find_filename(first, last); // algorithm defined elsewhere
	return wstring_view(filename, last - filename);
	}

class path
	{
public:
	// [...]
	path filename() const
		{	// parse the filename from *this and return a copy if present; otherwise,
			// return the empty path
		return parse_filename(native());
		}
	bool has_filename() const noexcept
		{	// parse the filename from *this and return whether it exists
		return !parse_filename(native()).empty();
		}
	// [...]
	};

In the std::experimental::filesystem implementation written before string_view, path::filename() contains the parsing logic, and returns a std::experimental::filesystem::path. has_filename is implemented in terms of filename, as depicted in the standard, allocating a path to immediately throw it away.

Iterator Debugging Support

In debugging builds, MSVC's string_view implementation is instrumented to detect many kinds of buffer management errors. The valid input range is stamped into string_view's iterators when they are constructed, and unsafe iterator operations are blocked with a message describing what the problem was.

// compile with cl /EHsc /W4 /WX /std:c++17 /MDd .program.cpp
#include <crtdbg.h>
#include <string_view>

int main() {
    // The next 3 lines cause assertion failures to go to stdout instead of popping a dialog:
    _set_abort_behavior(0, _WRITE_ABORT_MSG);
    _CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_FILE);
    _CrtSetReportFile(_CRT_ASSERT, _CRTDBG_FILE_STDOUT);

    // Do something bad with a string_view iterator:
    std::string_view test_me("hello world");
    (void)(test_me.begin() + 100); // dies
}
>cl /nologo /MDd /EHsc /W4 /WX /std:c++17 .test.cpp
test.cpp

>.test.exe
xstring(439) : Assertion failed: cannot seek string_view iterator after end

Now, this example might seem a bit obvious, because we're clearly incrementing the iterator further than the input allows, but catching mistakes like this can make debugging much easier in something more complex. For example, a function expecting to move an iterator to the next ')':

// compile with cl /EHsc /W4 /WX /std:c++17 /MDd .program.cpp
#include <crtdbg.h>
#include <string_view>

using std::string_view;

string_view::iterator find_end_paren(string_view::iterator it) noexcept {
    while (*it != ')') {
        ++it;
    }

    return it;
}

int main() {
    _set_abort_behavior(0, _WRITE_ABORT_MSG);
    _CrtSetReportMode(_CRT_ASSERT, _CRTDBG_MODE_FILE);
    _CrtSetReportFile(_CRT_ASSERT, _CRTDBG_FILE_STDOUT);
    string_view example{"malformed input"};
    const auto result = find_end_paren(example.begin());
    (void)result;
}
>cl /nologo /EHsc /W4 /WX /std:c++17 /MDd .program.cpp
program.cpp

>.program.exe
xstring(358) : Assertion failed: cannot dereference end string_view iterator
Pitfall #1: std::string_view doesn't own its data, or extend lifetime

Because string_view doesn't own its actual buffer, it's easy to write code that assumes data will live a long time. An easy way to demonstrate this problem is to have a string_view data member. For example, a struct like the following is dangerous:

struct X {
    std::string_view sv; // Danger!
    explicit X(std::string_view sv_) : sv(sv_) {}
};

because a caller can expect to do something like:

int main() {
    std::string hello{"hello"};
    X example{hello + " world"}; // forms string_view to string destroyed at the semicolon
    putc(example.sv[0]); // undefined behavior
}

In this example, the expression `hello + " world"` creates a temporary std::string, which is converted to a std::string_view before the constructor of X is called. X stores a string_view to that temporary string, and that temporary string is destroyed at the end of the full expression constructing `example`. At this point, it would be no different if X had tried to store a const char * which was deallocated. X really wants to extend the lifetime of the string data here, so it must make an actual copy.

There are of course conditions where a string_view member is fine; if you're implementing a parser and are describing a data structure tied to the input, this may be OK, as std::regex does with std::sub_match. Just be aware that string_view's lifetime semantics are more like that of a pointer.

Pitfall #2: Type Deduction and Implicit Conversions

Attempting to generalize functions to different character types by accepting basic_string_view instead of string_view or wstring_view prevents the intended use of implicit conversion. If we modify the program from earlier to accept a template instead of wstring_view, the example no longer works.

// program.cpp
// compile with: cl /std:c++17 /EHsc /W4 /WX
//    /I"%WindowsSdkDir%Include%UCRTVersion%cppwinrt" .program.cpp
#include <stddef.h>
#include <string.h>
#include <algorithm>
#include <iostream>
#include <locale>
#include <stdexcept>
#include <string>
#include <string_view>

#pragma comment(lib, "windowsapp")
#include <winrt/base.h>

template<class Char>
size_t count_letter_Rs(std::basic_string_view<Char> sv) noexcept {
    return std::count(sv.begin(), sv.end(),
        std::use_facet<std::ctype<Char>>(std::locale()).widen('R'));
}

int main() {
    std::wstring exampleWString(L"Hello wstring world!");
    winrt::hstring exampleHString(L"Hello HSTRING world!");
    count_letter_Rs(exampleWString); // no longer compiles; can't deduce Char
    count_letter_Rs(std::wstring_view{exampleWString}); // OK
    count_letter_Rs(exampleHString); // also no longer compiles; can't deduce Char
    count_letter_Rs(std::wstring_view{exampleHString}); // OK
}

In this example, we want exampleWString to be implicitly converted to a basic_string_view<wchar_t>. However, for that to happen we need template argument deduction to deduce CharT == wchar_t, so that we get count_letter_Rs. Template argument deduction runs before overload resolution or attempting to find conversion sequences, so it has no idea that basic_string is at all related to basic_string_view, and type deduction fails, and the program does not compile. As a result, prefer accepting a specialization of basic_string_view like string_view or wstring_view rather than a templatized basic_string_view in your interfaces.

In Closing

We hope string_view will serve as an interoperability bridge to allow more C++ code to seamlessly communicate. We are always interested in your feedback. Should you encounter issues please let us know through Help > Report A Problem in the product, or via Developer Community. Let us know your suggestions through UserVoice. You can also find us on Twitter (@VisualC) and Facebook (msftvisualcpp).

Experiencing Data Access Issue in Azure and OMS portal for Log Analytics in South East Australia- 08/22 – Mitigated

Final Update: Wednesday, 22 August 2018 02:51 UTC

We've confirmed that all systems are back to normal with no customer impact as of 08/22, 01:00 UTC. Our logs show the incident started on 08/21, 23:00 UTC and that during the 2 hours that it took to resolve the issue 15% of customers experienced intermittent data access issues.
  • Root Cause: The failure was due to an issue in one of our dependent platform services..
  • Incident Timeline: 2 Hours - 08/21, 23:00 UTC through 08/22, 01:00 UTC

We understand that customers rely on Azure Log Analytics as a critical service and apologize for any impact this incident caused.

-Leela


MySQL with Entity Framework Core in ASP.NET Core

.NET Core offers.NET developers a great way to create cross-platform applications. If you want to create a web app with some database support, most likely you will use Entity Framework (EF) Core and ASP.NET Core. In this blog post, I'd like to give you some tips if you choose to use MySQL.

1. Db Provider choice

Although there're a few MySQL providers to choose from in the official EF Core Docs site, I would recommend the most popular Pomelo.EntityFrameworkCore.MySql. I tried other providers and it gave me issues here and there.

Specifically, you can use the following code to add a DbContext in your service configuration:

services.AddCors();

services.AddDbContext<KnowledgeContext>(options => options.UseMySql(Configuration["MySqlConnStr"]));

services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);

And don't forget to pass the DbContext to your controller's constructor.

2. Use Fluent API instead of Data Annotation

This is more of an EF Core tip than a MySQL one. Certain restrictions such as foreign keys can only be specified with  the Fluent API.

3. Where to call Database.EnsureCreated

Again, an EF Core tip. DbContext.Database.EnsureCreated() is new method to ensure that the database and the required schema for the context exist. The database and the schema will be created if they don't exist. However, in ASP.NET Core, every time a request comes, a controller's constructor is called. So you probably want to put the EnsureCreated call in the Startup class. Please refer to https://stackoverflow.com/questions/36958318/where-should-i-put-database-ensurecreated.

Microsoft Azure Service Fabric 6.3 Refresh Release (CU1) Notes

Hi all,

Today a new refresh of Service Fabric is being released. This is the first refresh release for 6.3 (CU1) and it contains updates to the following packages:

  • Service Fabric Runtime
    • Ubuntu - 6.3.124
    • Windows - 6.3.176.9494
    • Service Fabric for Windows Server Service Fabric Standalone Installer Package - 6.3.176.9494
  • .NET SDK
    • Windows .NET SDK - 3.2.176
    • Microsoft.ServiceFabric - 6.3.176
    • Reliable Services and Reliable Actors - 3.2.176
    • ASP.NET Core Service Fabric integration - 3.2.176

 

Please find the release notes here with more information and a full list of current packages in this release:
Microsoft-Azure-Service-Fabric-Release-Notes-SDK-3.2-CU3-Runtime-6.3-CU1

Today the .NET SDK and Windows runtime is available through Web Platform Installer, NuGet packages. Ubuntu runtime is available on the apt-get repositories. On Mac devices, there is a container image available which contains the Ubuntu runtime. Direct download links are in the attached release notes.

In Azure, the updated version is rolling through regions. This typically takes 5-7 days to complete as we upgrade clusters in the regions in a rolling fashion. As regions complete, the new version will also be available for manual upgrades and new clusters.

To raise issues, please look here:https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-support

Please use the comments below if you have any questions to the release notes and information provided.

Thanks,

The Service Fabric Team


Hosted Agent Issues in West Europe – 08/22 – Mitigated

Final Update: Wednesday, August 22nd 2018 19:19 UTC

The fix was rolled out and our telemetry indicates that the impact has been mitigated as of 19:00 UTC. This incident has impacted the users of Hosted VS2017 pool in West Europe region. Users should no longer see any delays or disconnect failures requesting a hosted agent for builds and releases. The existing backlog in the queue is being processed and we expect to be fully caught up within the next 30 minutes.

Sincerely,
Sri Harsha


Update: Wednesday, August 22nd 2018 17:50 UTC

Our DevOps team continues to investigate issues with Hosted Agent for Build and Release. Changes to our service got approved and we are working on rolling out the potential fix. After the rollout, the service will need additional time for the queue to be fully processed, during that time users may continue to see delays/errors. The problem began at 08:30 UTC . We currently have no estimated time for resolution.

  • Work Around: If possible, using Private Agents for Build & Release works around this issue.
  • Next Update: Before Wednesday, August 22nd 2018 22:00 UTC

Sincerely,
Ariel Jacob


Update: Wednesday, August 22nd 2018 15:50 UTC

Our DevOps team continues to investigate issues with Hosted Agent for Build and Release, both delays in assigning the request to an agent and in disconnect failures when finally assigned. Root cause is not fully understood at this time. We are in the process of making some changes to our service which we believe will help with the delays. The problem began at 08:30 UTC . We currently have no estimated time for resolution.

  • Work Around: If possible, using Private Agents for Build & Release works around this issue
  • Next Update: Before Wednesday, August 22nd 2018 18:00 UTC

Sincerely,
Ariel Jacob


Update: Wednesday, August 22nd 2018 12:40 UTC

Our DevOps team continues to investigate issues with Hosted Agent for Build and Release, both delays in assigning the request to an agent and in disconnect failures when finally assigned. Root cause is not fully understood at this time. We are continuing to investigate a possible issue with our database, but are also investigating other avenues as well. The problem began at 08:30 UTC . We currently have no estimated time for resolution.

  • Work Around: If possible, using Private Agents for Build & Release works around this issue
  • Next Update: Before Wednesday, August 22nd 2018 16:00 UTC

Sincerely,
Dexter


Update: Wednesday, August 22nd 2018 10:51 UTC

Our DevOps team continues to investigate issues with Build Delays in West Europe. Root cause is not fully understood at this time, though it is suspected to be related to one of our back-end databases. The problem began at 8:30 UTC. We currently have no estimated time for resolution.

  • Next Update: Before Wednesday, August 22nd 2018 13:00 UTC

Sincerely,
Ariel Jacob


Initial notification: Wednesday, August 22nd 2018 09:53 UTC

We're investigating delays for Build in West Europe.
We are seeing increased times in acquiring build machines, this may lead to delays for some customers.
Customers may see long build times or timeouts.

  • Next Update: Before Wednesday, August 22nd 2018 11:00 UTC

Sincerely,
Ariel Jacob

ASP.NET Core 2.2.0-preview1 now available

Today we’re very happy to announce that the first preview of the next minor release of ASP.NET Core and .NET Core is now available for you to try out. We’ve been working hard on this release over the past months, along with many folks from the community, and it’s now ready for a wider audience to try it out and provide the feedback that will continue to shape the release.

How do I get it?

You can download the new .NET Core SDK for 2.2.0-preview1 (which includes ASP.NET 2.2.0-preview1) from https://www.microsoft.com/net/download/dotnet-core/2.2.

Visual Studio requirements

Customers using Visual Studio should also install and use the Preview channel of Visual Studio 2017 (15.9 Preview 1 at the time of writing) in addition to the SDK when working with .NET Core 2.2 and ASP.NET Core 2.2 projects.

Azure App Service Requirements

If you are hosting your application on Azure App Service, you should follow these instructions to install the site extension for hosting your 2.2.0-preview1 applications.

Impact to machines

Please note that is a preview release and there are likely to be known issues and as-yet-to-be discovered bugs. While the .NET Core SDK and runtime installs are side-by-side, your default SDK will become the latest one. If you run into issues working on existing projects using earlier versions of .NET Core after installing the preview SDK, you can force specific projects to use an earlier installed version of the SDK using a global.json file as documented here. Please log an issue if you run into such cases as SDK releases are intended to be backwards compatible.

What’s new in 2.2

We’re publishing a series of posts here that go over some of the new feature areas in detail. We’ll update the post with links to these posts as they go live over the coming days:

  • API Controller Conventions
  • Endpoint Routing
  • Health Checks
  • HTTP/2 in Kestrel
  • Improvements to IIS hosting
  • SignalR Java client

In addition to these features area, we’ve also:

  • Updated our web templates to Bootstrap 4 and Angular 6

You can see these new features in action in our most recent ASP.NET Community Standup.

For a detailed list of all features, bug fixes, and known issues refer to our release notes.

Migrating an ASP.NET Core 2.1 project to 2.2

To migrate an ASP.NET Core project from 2.1.x to 2.2.0-preview1, open the project’s .csproj file and change the value of the the <TargetFramework> element to netcoreapp2.2. You do not need to do this if you’re targeting .NET Framework 4.x.

Giving Feedback

The main purpose of providing previews is to solicit feedback so we can refine and improve the product in time for the final release. Please help provide us feedback by logging issues in the appropriate repository at https://github.com/aspnet or https://github.com/dotnet. We look forward to receiving your feedback!

WebJobs in Azure with .NET Core 2.1

WebJobs aren't new to Azure or .NET. There's even a default Azure WebJob template in Visual Studio 2017 for the full .NET Framework. However, a similar template for WebJobs in .NET Core is somehow missing from Visual Studio. In this post, I'm using .NET Core 2.1.

Creating a WebJob in .NET Core isn't hard, but you have to know some tricks, especially if you want to use some .NET Core goodies like logging and DI.

In this post, we're going to build a WebJob and release it to Azure using Visual Studio, the Azure portal, and VSTS.

You can find the code samples for this post on GitHub.

What are WebJobs

A WebJob is a program running in the background of an App Service. It runs in the same context as your web app at no additional cost. Maybe you need to do some hourly task or do some cleanup task every night at 1 AM. Azure Application Insights uses a WebJob to report your app's statistics.

WebJobs can be scheduled, like hourly or daily, but they can also be triggered. A trigger could be a file upload or a new message on a queue.

WebJobs vs. Functions

I've often found comparisons between WebJobs and Azure Functions. In a way, Functions are the successors to WebJobs. Functions are (usually) small pieces of code that run in Azure and are, just like WebJobs, triggered at a certain event, including an HTTP trigger.

Functions are often a great alternative to WebJobs, but if you already have a web app it could make sense to use a WebJob instead. Especially if you want to share code and/or settings between the WebJob and your web app as they run in the same context, which also makes deployment quite easy.

Creating a Storage Account

Before we continue let's take care of something first. A WebJob requires an Azure Storage Account.  I'll quickly walk you through the process of creating one.

In Azure, find "Storage Accounts" and add one. You'll have to pick a name that's unique across Azure. Other than that you can leave the defaults. We're talking about cents per GB, so don't worry about costs too much.

Once your Storage Account is ready, select it and find your "Access keys". We'll need one of the two connection strings later.

Creating a WebJob

As said, there's a WebJob template for the full .NET Framework. I recommend you check it out. Start by creating an ASP.NET Web Application and then add a new WebJob. If you try to create the WebJob right away you'll get an error saying that the project needs to be saved first (although it does create the WebJob).

We're here for the .NET Core version of a WebJob though. So start by creating an ASP.NET Core Web Application and then add a new .NET Core Console App project to your solution.

The first thing we need to do to is install the Microsoft.Azure.WebJobs package from NuGet. We should also install Microsoft.Azure.WebJobs.Extensions. Here's the catch though, the latest stable versions of these libraries have dependencies on the full .NET Framework so we're going to need version 3.0.0-beta5 (at the time of this writing), which is fully compatible with .NET Core.

Other NuGet packages we'll need are Microsoft.Extensions.Options.ConfigurationExtensions (which also gives us the Microsoft.Extensions.Options package, which we also need), Microsoft.Extensions.DependencyInjection and Microsoft.Extensions.Logging.Console. Be sure to install version 2.1.0 of these packages because there seems to be a bug in .NET Core 2.1 that prevents you from using packages with patch versions, like 2.1.1.

Join the Program

The next thing we need to do is change our Program.cs file. If you've created a WebJob using the .NET Framework template you can simply copy and paste the Program.cs file that was generated there (although you might want to change the namespace).

using Microsoft.Azure.WebJobs;

namespace NetCoreWebJob.WebJob
{
    // To learn more about Microsoft Azure WebJobs SDK, please see https://go.microsoft.com/fwlink/?LinkID=320976
    internal class Program
    {
        // Please set the following connection strings in app.config for this WebJob to run:
        // AzureWebJobsDashboard and AzureWebJobsStorage
        private static void Main()
        {
            var config = new JobHostConfiguration();
            if (config.IsDevelopment)
            {
                config.UseDevelopmentSettings();
            }
            var host = new JobHost(config);
            // The following code ensures that the WebJob will be running continuously
            host.RunAndBlock();
        }
    }
}

Adding Configuration and DI

So I promised you'd get all the .NET Core goodies like logging and DI. By default, a Console App doesn't have any of that, but you can add it yourself.

private static void Main()
{
    IServiceCollection services = new ServiceCollection();
    ConfigureServices(services);

    // ...
}

private static IConfiguration Configuration { get; set; }

private static void ConfigureServices(IServiceCollection services)
{
    var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

    Configuration = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.{environment}.json", optional: true, reloadOnChange: true)
        .AddEnvironmentVariables()
        .Build();

    services.AddSingleton(Configuration);
    services.AddTransient<Functions, Functions>();
    services.AddLogging(builder => builder.AddConsole());
}

Next, create an appsettings.json file and set the "Copy to Output Directory" property to "Copy always". The appsettings.json file should have two connection strings as mentioned in the Program.cs template file. These are the Storage Account connection strings we created earlier.

{
  "Logging": {
    "LogLevel": {
      "Default": "Warning"
    }
  },
  "ConnectionStrings": {
    "AzureWebJobsDashboard": "[your Storage Account connection string]",
    "AzureWebJobsStorage": "[your Storage Account connection string]"
  }
}

The next thing we need is a custom IJobActivator that can be used to inject dependencies into our classes. It needs to be set on the JobHostConfiguration in the Program class.

using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.DependencyInjection;
using System;

namespace NetCoreWebJob.WebJob
{
    public class JobActivator : IJobActivator
    {
        private readonly IServiceProvider services;

        public JobActivator(IServiceProvider services)
        {
            this.services = services;
        }

        public T CreateInstance<T>()
        {
            return services.GetService<T>();
        }
    }
}
var config = new JobHostConfiguration();
config.JobActivator = new JobActivator(services.BuildServiceProvider());

Adding a Trigger

After that, create a class and name it Functions (just like in the WebJob template). The Functions class will have the actual code for our WebJob.

Of course, we'll need to add a trigger. This is different than the full .NET Framework. After all, the template uses a static method, which makes DI impossible. Speaking of DI, notice that we've also added the Functions class itself to the DI container.

For simplicity, we'll use a TimerTrigger, which is triggered with a so-called CRON expression. This simply means it's triggered at a certain minute, hour, day, etc. In this example, it triggers every minute.

We'll also need to configure timers on the JobHostConfiguration.

using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using System;

namespace NetCoreWebJob.WebJob
{
    public class Functions
    {
        private readonly ILogger<Functions> logger;

        public Functions(ILogger<Functions> logger)
        {
            this.logger = logger;
        }

        public void ProcessQueueMessage([TimerTrigger("* * * * *")]TimerInfo timerInfo)
        {
            logger.LogInformation(DateTime.Now.ToString());
        }
    }
}
var config = new JobHostConfiguration();
config.JobActivator = new JobActivator(services.BuildServiceProvider());
config.UseTimers();

Running the example

If you did everything correctly, or if you're running my code from GitHub, you should now be able to run the Console App. If you break on exceptions or if you're watching the Output window you may notice a lot of StorageExceptions. Don't worry about them and ignore them. This is a bug in the WebJobs library and will not affect your program. It may take a minute for your trigger to go off, so have a little patience.

If you head over to your Azure Storage Account you should see two Blob Containers, "azure-jobs-host-output" and "azure-webjobs-hosts". There's quite a lot going on here, but you can just ignore it. I've found that my WebJob triggers wouldn't go off for some reason, deleting the Blob Containers usually helped. Apparently, there's some state stored in there which isn't always disposed of properly when (re-)adding and removing WebJobs.

Deploying to Azure

The next thing we want to do is deploy our WebJob to Azure. In order for a WebJob to run it needs some executable script that it can use to get going. Many file types are supported, but for us Windows users it makes sense to use an exe, cmd, bat or PowerShell file.

A Console App used to be an exe file, but in .NET Core, it produces a regular DLL file that we need to start manually. So, create a file and name it "run.cmd" and make sure that it's encoded in UTF-8 no BOM (you can check this using something like Notepad++). It just needs a single line of code, which is "dotnet NetCoreWebJob.WebJob.dll". This runs your Console App. Make sure you set the "Copy to Output Directory" of the file to "Copy always".

One last thing, for some reason Azure WebJobs needs all the dependencies of a WebJob, which means all .NET Core packages we used to build it. You can do this by editing the csproj file and adding "<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>" to the first <PropertyGroup> (underneath "<TargetFramework>").

Before we can deploy our WebJob we need to deploy our web app. Right-click the ASP.NET project and click "Publish...". Simply follow the wizard and Visual Studio will deploy your app for you. You can create a new web app or select an existing one. This step isn't strictly necessary as you can host stand-alone WebJobs, but this should be familiar and it gives you an App Service we can use for our WebJob.

Deploy using Visual Studio

Deploying WebJobs using Visual Studio should be easy as pie. In fact, you probably already know how to do this (don't do it yet though). Right-click your WebJob project and click "Publish...". The following wizard looks a whole lot like the publication of a web app, which we just did. You can pick "Select existing" and pick the Azure web app we just created.

Unfortunately, Microsoft messed up this feature in the worst way possible. Visual Studio will deploy the WebJob with the same name as the project, which is "NetCoreWebJob.WebJob", except the dot is an illegal character in a WebJob name! This messed up my project so bad I had to manually edit it to make my solution working again. Nice one, Microsoft!

So here's what you do. At the start of the wizard, where you pick either a new or existing App Service, click the arrow next to "Publish immediately" and pick "Create Profile" instead. Now you can first change the name of your WebJob in the settings and deploy after that. Make sure you don't select "Remove additional files at destination"  or you'll remove your web app.

Now, browse to the Azure Portal and look up your web app. You'll find "WebJobs" in the menu. You'll see your WebJob, but it's not doing anything. You need to manually run it by selecting it and clicking "Run". The status should update to "Running". You can now check out the logs to see that it actually works. You may see an error about the connection strings, but you can ignore those. If you toggle the output you'll still see a log is written to the console which lets you know it works! If you don't see a log right away try waiting a minute or two and don't forget to manually refresh the output.

Deploy using the Azure Portal

When you add a new WebJob you'll need to fill out some options. You can make up some name, set the type to "Triggered" and the triggers to "Manual". Your alternatives are a "Continuous" WebJob, which just runs and closes (unless you've implemented an endless loop in your application); and a "Scheduled" triggered job, which is basically what we have except we've implemented the schedule ourselves.

The "File upload" needs a bit of explanation. Here you can upload a zip file that contains your WebJob. So head over to Visual Studio and build your solution. Then go to the output folder of your WebJob project, something like "MyProjectbin[Debug|Release]netcoreapp2.1", and put everything in that folder into a zip file. Then select it in the "File upload" in your new WebJob.

It will take a few seconds for Azure to create your WebJob so keep refreshing until it pops up. After that, you have to start it manually again and you can check out the logs.

Deploy using VSTS

Ultimately, we want to add our WebJob to our CI/CD pipeline in VSTS. Unfortunately, this functionality doesn't come out of the box. Luckily, it's not very difficult either. If you're not familiar with builds or releases in VSTS check out one of my previous blogs Azure Deployment using Visual Studio Team Services (VSTS) and .NET Core or ARM Templates to Deploy your Resources to Azure.

When you're in the Azure Portal find the "App Service Editor (Preview)" of your App Service. This lets you browse all the files in your App Service. One thing we notice is that your WebJob is located in "App_Datajobstriggered[WebJob name]". And since your WebJob is really just the output of the WebJob project build it's simply a matter of copying your WebJob files to App_Data.

The build

So go to VSTS and create a new build. Select your repository, your branch, and select "ASP.NET Core" as a template. We only need to change two things here. We need to change the existing "Publish" task and add a new ".NET Core" task to publish our WebJob.

Change the name of the existing publish task to "Publish web app", untick the "Publish Web Projects" checkbox, and enter the "Path to project(s)", which is "**/NetCoreWebJob.csproj". Also, untick the "Zip Published Projects" and "Add project name to publish path" checkboxes as they will eventually mess up our release.

After that create a new .NET Core task, set the "Command" to "publish" and change the name of the task to "Publish web job". Again, untick "Publish Web Projects" and set the "Path to project(s)", which is "**/NetCoreWebJob.WebJob.csproj". And once again, don't zip the published projects or add the project name to the publish path. The last step here is the "Arguments" field, which can be copy/pasted from the other publish step, except we're going to add a bit to it: "--configuration $(BuildConfiguration) --output $(build.artifactstagingdirectory)App_DatajobstriggeredWebJobVSTS".

The release

Last, but not least, is the release. Create a new release in VSTS, pick the "Azure App Service deployment" template and fill in the blanks, which is an artifact and your Azure settings in the Environment. Because we're not zipping our build we just need to change one setting. In the "Deploy Azure App Service" task is a "Package or folder" setting, which has a default of "[...]/*.zip" which obviously isn't going to work. Instead, use the browser (the button with "...") and select your drop folder.

Save it, hit new release and pick your latest build. If all goes well you should see your new WebJob in the Azure Portal!

Wrap up

Hopefully, Microsoft will come with a built-in solution for creating, consuming, and deploying WebJobs to Azure in .NET Core soon. Until that time it's not particularly difficult, it's just a matter of knowing what to do.

In this post, we've seen the TimerTrigger, but there's a QueueTrigger, a BlobTrigger, and a FileTrigger as well. Experiment, google, and read the official documentation.

Don't forget to delete your resources.

Happy coding!

Issue with security update for the Remote Code Execution vulnerability in SQL Server 2016 SP1 (GDR): August 14, 2018

On Tuesday August 14 we published a Security Update for six different releases of SQL Server 2016 and 2017. For one of those releases, SQL Server 2016 SP1 GDR (KB4293801), an issue may occur after applying the update where the sqlceip.exe process experiences an unhandled exception.  This will only occur if the updated instance was currently configured to collect SQL  Customer Experience Improvement Program (CEIP) information. This does not impact the operation of the updated SQL Server engine, however this may impact a SQL Server Failover Cluster Instance node if configured for CEIP.

Please note that this issue only impacts the SQL Server 2016 SP1 GDR (KB4293801) release, which is for SQL Server 2016 SP1 instances that have not had any CU updates applied.

Therefore, the update has been replaced. If you have previously applied the original update KB4293801, we recommend that you install KB4458842 as soon as possible.

You may optionally first uninstall KB4293801 , but it is not necessary as KB4458842 supersedes and replaces KB4293801.

Measuring Model Goodness – Part 1

Author: Dr. Ajay Thampi, Lead Data Scientist

Data and AI are transforming businesses worldwide from finance, manufacturing and retail to healthcare, telecommunications and education. At the core of this transformation is the ability to convert raw data into information and useful, actionable insights. This is where data science and machine learning come in.

Building a good and reliable machine learning system requires us to be methodical by:

  • understanding the business needs
  • acquiring and processing the relevant data
  • accurately formulating the problem
  • building the model using the right machine learning algorithm
  • evaluating the model, and
  • validating the performance in the real world before finally deploying it

This entire process has been documented as the Team Data Science Process (TDSP) at Microsoft, captured in the figure below.

Image may be NSFW.
Clik here to view.

This post, as part of a two-part series, is focused on measuring model goodness, specifically looking at quantifying business value and converting typical machine learning performance metrics (like precision, recall, RMSE, etc.) to business metrics. This is typically how models are validated and accepted in the real world. The relevant stages of the process are highlighted in red in the figure above. Measuring model goodness also involves comparing the model performance to reasonable baselines, which will also be covered in this post. To illustrate all of this, two broad classes of machine learning problems will be looked at:

  • Classification: Covered in Part 1
  • Regression: To be covered in Part 2

Classification

Classification is the process of predicting qualitative or categorical responses [source]. It is a supervised machine learning technique that classifies a new observation to a set of discrete categories. This is done based on training data that contains observations whose categories are known. Classification problems can be binary in which there are two target categories or multi-class in which there are more than two mutually exclusive target categories.

In this post, I will look at the binary classification problem taking breast cancer detection as a concrete example. I will be using the open breast cancer Wisconsin dataset for model building and evaluation. The techniques discussed below can easily be extended to other binary and multi-class classification problems. All the source code used to build and evaluate the models can be found on Github here.

Let’s first understand the business context and the data. Breast cancer is the most common cancer in women and is the main cause of death from cancer among women [source]. Early detection of the cancer type — whether it is benign or malignant — can help save lives by picking the appropriate treatment strategy. In this dataset, there are 569 cases in total out of which 357 are benign (62.6%) and 212 are malignant (37.4%). There are 30 features in total, computed from a digitised image of a biopsy of a breast mass. These features describe the characteristics of the nuclei like radius, texture, perimeter, smoothness, etc. The detection of the type of cancer is currently being done by a radiologist which is time consuming and the business need is to speed up the diagnosis process so that the appropriate treatment can be started early. Before diving too deep into the modelling stage, we need to understand the value/cost of making the right/wrong prediction. This information can be represented using a value-cost matrix as shown below:

Image may be NSFW.
Clik here to view.

The rows in the matrix represent the actual class and the columns represent the predicted class. The first row/column represents in the negative class (benign in this case) and the second row/column represents the positive class (malignant in this case). Here's what each of the elements in the matrix represents:

  • Row 0, Column 0: Value of predicting the negative class correctly
  • Row 0, Column 1: Cost of predicting the positive class wrongly
  • Row 1, Column 0: Cost of predicting the negative class wrongly
  • Row 1, Column 1: Value of predicting the positive class correctly

For breast cancer detection, we can define the value-cost matrix as follows:

Image may be NSFW.
Clik here to view.

  • Detecting benign and malignant cases correctly are given equal and positive value. Although the malignant type is the worse situation to be in for the patient, the goal is to diagnose early, start treatment and cure both types of cancer. So, from a treatment perspective, detecting both cases accurately have equal value.
  • The cost of flagging a malignant case as benign (false negative) is much worse than flagging a benign case as malignant (false positive). Therefore, the false negative is given a cost of -4 and the false positive is given a cost of -2.

Another business need is to automate as much of the process as possible. One way of doing this is to use the machine learning model as a filter, to automate the detection of simpler benign cases and only flag the possible malignant cases for the radiologist to review. This is shown in the diagram below. We can quantify this business need by looking at the % of cases to review manually (denoted as x in the figure). Ideally, we want to automate everything without requiring the human in the loop. This means that the model should have 100% accuracy and x should be equal to the actual proportion of positive classes.

Image may be NSFW.
Clik here to view.

After the business needs are properly quantified, the next step is to define reasonable baselines to compare our models to.

  • Baseline 1: Random classifier that decides randomly if the cancer type is benign / malignant
  • Baseline 2: Majority-class classifier that picks the majority class all the time. In this case, the majority class is benign. This strategy would make much more sense if the classes were heavily imbalanced.

We could add a third baseline which is the performance of the radiologist or any other model that the business has currently deployed. In this example though, this information is not known and so this baseline is dropped.

Now comes the fun part of building the model. I will gloss over a lot of the detail here, as an entire post may be necessary to go through the exploratory analysis, modelling techniques and best practices. I used a pipeline consisting of a feature scaler, PCA for feature reduction and finally a random forest (RF) classifier. 5-fold cross validation and grid search were done to determine the optimum hyperparameters. The entire source code can be found here.

Once the models are trained, the next step is to evaluate them. There are various performance metrics that can be used but, in this post, I will go over the following 6 metrics as they can very easily be explained, visualised and translated into business metrics

  • True Positives (TP) / True Positive Rate (TPR): Number of correct positive predictions / Probability of predicting positive given that the actual class is positive
  • False Negatives (FN) / False Negative Rate (FNR): Number of wrong negative predictions / Probability of predicting negative given that the actual class is positive
  • True Negatives (TN) / True Negative Rate (TNR): Number of correct negative predictions / Probability of predicting negative given that the actual class is negative
  • False Positives (FP) / False Positive Rate (FPR): Number of wrong positive predictions / Probability of predicting positive given that the actual class is negative
  • Precision (P): Proportion of predicted positives that are correct
  • Recall (R): Proportion of actual positives captured

All these metrics are inter-related. The first four can be easily visualised using a ROC curve and a confusion matrix. The last two can be visualised using a precision-recall (PR) curve. Let's first visualise the performance of the model and the baselines using the ROC and PR curves.

Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.

Both plots show the performance of the models using various thresholds to determine the positive and negative classes. As can be seen, the RF classifier outperforms the baselines in every respect. Once the appropriate threshold is picked, we can plot the normalised confusion matrices as shown below. These matrices show the conditional probabilities of predicted values given the actual value. We can see that the baselines do quite poorly especially in predicting the positive class — the false negative rates are quite high. The RF classifier, on the other hand, seem to get much more of the positive and negative class predictions right achieving a TPR of 93% and TNR of 97%.

Image may be NSFW.
Clik here to view.

Now that we’ve established that the new RF model outperforms the baselines, we still need to make sure that it meets our business objectives. i.e. high positive business value and fewer cases to review manually. We therefore need to translate the above performance metrics into the following:

  • Total expected value/cost of the model
  • % of cases that need to be reviewed manually

The first business metric can be computed by taking the dot product of the flattened confusion matrix (normalised by the total number of samples, S) with the flattened value-cost matrix. This is shown below.

Image may be NSFW.
Clik here to view.

Note that S is a scalar equal to TN + FP + FN + TP. The expected value is essentially the weighted average of the values/costs, where the weights are the probabilities of each of the four outcomes [source].

The second business metric can be computed using precision and recall as follows.

Image may be NSFW.
Clik here to view.

The positive class rate is known from the data used to evaluate the model, i.e. the test set. Based on the business needs, we then need to decide how much of those positive classes we must accurately determine, i.e. the recall. If we want to detect all the positive cases, the target recall should be 100%. We can then find the corresponding precision (and threshold) from the PR curve. For the ideal model with 100% precision and recall, the proportion of positive cases to review would be equal to the actual positive class rate. This means we could, in theory, achieve 100% automation without a human in the loop.

The following two plots show the final evaluation of the model goodness in terms of the two key business metrics.

Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.

Observations:

  • Both baseline models have quite large costs because of the high false negatives. The random forest classifier has a good positive value by getting a lot of the positive and negative cases right.
  • For the case of breast cancer detection, we want to capture all the positive/malignant cases, i.e. 100% recall. Looking at 50% or 75% recall does not make sense in this example because of the high cost of not treating the malignant cases. For other binary classification problems like fraud detection for instance, a lesser recall may be acceptable so long as the model can flag the high cost-saving fraudulent cases. Regardless, we can see that the RF classifier outperforms the baselines on the automation front as well. To capture 100% of the malignant cases, the radiologist would only need to review about 50% of all cases, i.e. 13% additional false positives, whereas he/she would have to review almost all the cases using the baseline models.

In conclusion, the first part of this series has looked at measuring model goodness through the lens of the business, specifically looking at classification-type problems. For further details on this subject, the books below are great references. In the second and final part of this series, I will cover regression which requires looking at slightly different metrics.

Further Reading

Viewing all 35736 articles
Browse latest View live