We are happy to announce today that the Windows Compatible Product List is now LIVE!

Features that are available in this release

• Search functionality across all “Certified” devices and systems for our entire catalog.
• Customers needing an OEM System report for RFPs can search for the product/marketing name and receive a downloadable Verification Report. *Systems do not support a D&U status. Therefore, that search dropdown must be left set to All for results to show.
• D & U status is displayed for all Device Classes.

How search works for Marketing/Product Name

• Searches should be limited to plain text without quotes or wildcards.
• Each space between a word is equivalent to an AND expression.
• No special search limiting terms should be used. I.E. do not use (OR, AND, ””, *, ?, etc.)
• Searches within this area will find matches in our product name , submission name, and marketing name fields.

How search works for Company Name

• Searches should be limited to plain text without quotes or wildcards (same as above)
• We will auto-complete found results after 5 letters are typed.Some companies are listed twice. This is a result of merging our Sysdev database accounts with Hardware Dev Center. Each entity is different, so it’s best to just search using the Marketing/Product Name field.

Register today for these workshops, designed to give attendees a hands-on view of the scalable Data and AI solutions on Azure. The contents of these 3-day workshops will align to a solution-based approach on how customers look at their Data + AI stack, what challenges they face, and how Azure’s services cater to those challenges.

Date: 30 January - 1 February, 2019
Location: Sydney Harbour Marriott
Time: 08:30-17:00 each day
Cost: $899 AUD

What is Microsoft 365 Business?

Microsoft 365 Business is an integrated solution that brings together the best-in-class productivity of Office 365 with advanced security and device management capabilities to help safeguard small and medium-sized businesses. Microsoft 365 Business positions you to grow with new services and customers, increase customer lifetime value, and differentiate your offerings. Learn more at aka.ms/m365bpartners

Now, AI is one of important technologies.
Almost all platforms have API sets of AI. Following list is technology names per platform.

• Windows 10: Windows ML
• Android: TensorFlow
• iOS: CoreML

Xamarin can call native API sets using C#. It means you can implement AI feature on your app using Xamarin. This article will be introducing how to use AI APIs with Xamarin.Forms.

Create a project

Open Visual Studio 2017, then create a new project that is Mobile App (Xamarin.Form) of Cross-Platform category. And then select Blank, select Android, iOS, Windows(UWP) and .NET Standard.

Add base feature that exclude AI to here. I'll use a take photo feature in this app. So, add Xam.Plugin.Media NuGet package to all projects, then setup projects in accordance with the readme file shown. And then edit MainPage.xaml like below:

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage
    x:Class="AIApp.MainPage"
    xmlns="http://xamarin.com/schemas/2014/forms"
    xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
    xmlns:ios="clr-namespace:Xamarin.Forms.PlatformConfiguration.iOSSpecific;assembly=Xamarin.Forms.Core"
    xmlns:local="clr-namespace:AIApp"
    Title="Safe Area"
    ios:Page.UseSafeArea="True">

    <StackLayout>
        <Image
            x:Name="picture"
            Aspect="AspectFill"
            VerticalOptions="FillAndExpand" />
        <Label x:Name="output" HorizontalOptions="CenterAndExpand" />
        <StackLayout Orientation="Horizontal">
            <Button
                Clicked="PickPhotoButton_Clicked"
                HorizontalOptions="FillAndExpand"
                Text="Pick a picture" />
            <Button
                Clicked="TakePhotoButton_Clicked"
                HorizontalOptions="FillAndExpand"
                Text="Take a picture" />
        </StackLayout>
    </StackLayout>

</ContentPage>

At next, edit the code behind like below:

using Plugin.Media;
using Plugin.Media.Abstractions;
using System;
using System.Threading.Tasks;
using Xamarin.Forms;

namespace AIApp
{
    public partial class MainPage : ContentPage
    {
        public MainPage()
        {
            InitializeComponent();
        }

        private async void TakePhotoButton_Clicked(object sender, EventArgs e)
        {
            await ProcessPhotoAsync(true);
        }

        private async void PickPhotoButton_Clicked(object sender, EventArgs e)
        {
            await ProcessPhotoAsync(false);
        }

        private async Task ProcessPhotoAsync(bool useCamera)
        {
            await CrossMedia.Current.Initialize();
            if (useCamera ? !CrossMedia.Current.IsTakePhotoSupported : !CrossMedia.Current.IsPickPhotoSupported)
            {
                await DisplayAlert("Info", "Your phone doesn't support photo feature.", "OK");
                return;
            }

            var photo = useCamera ? 
                await CrossMedia.Current.TakePhotoAsync(new StoreCameraMediaOptions()) : 
                await CrossMedia.Current.PickPhotoAsync();
            if (photo == null)
            {
                picture.Source = null;
                return;
            }

            picture.Source = ImageSource.FromFile(photo.Path);

            var service = DependencyService.Get<IPhotoDetector>();
            if (service == null)
            {
                await DisplayAlert("Info", "Not implemented the feature on your device.", "OK");
                return;
            }

            using (var s = photo.GetStream())
            {
                var result = await service.DetectAsync(s);
                output.Text = $"It looks like a {result}";
            }
        }
    }
}

In this code, using IPhotoDetector interface to detect a photo. The interface is just a method that is DetectAsync.

using System.IO;
using System.Threading.Tasks;

namespace AIApp
{
    public interface IPhotoDetector
    {
        Task<FriesOrNotFriesTag> DetectAsync(Stream photo);
    }

    public enum FriesOrNotFriesTag
    {
        None,
        Fries,
        NotFries,
    }
}

Create ML models

I use Microsoft Cognitive Services Custom Vision(https://customvision.ai) to create ML models. Create Fries and NotFries tags on the project of Custom Vision.
Custom Vision service has a feature that generate CoreML, TensorFlow and ONNX files. Please read the following document to know more information.

Export your model for use with mobile devices | Microsoft Docs

The point is that select General (compact) of Domains category when creating project.

After training the model, you can export the ML models from Export button at Performance tab.

Add Windows 10 implementation

Windows 10 has Windows ML feature.

Windows Machine Learning | Microsoft Docs

Add the onnx file to Assets folder on the UWP project, then generated a C# file for use the onnx file.

Add PhotoDetector.cs file to UWP project, and then edit the file like below:

using System;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Windows.AI.MachineLearning;
using Windows.Graphics.Imaging;
using Windows.Media;
using Windows.Storage;
using Xamarin.Forms;

[assembly: Dependency(typeof(AIApp.UWP.PhotoDetector))]
namespace AIApp.UWP
{
    public class PhotoDetector : IPhotoDetector
    {
        private FriesOrNotFriesModel _model;
        public async Task DetectAsync(Stream photo)
        {
            await InitializeModelAsync();
            var bitmapDecoder = await BitmapDecoder.CreateAsync(photo.AsRandomAccessStream());
            var output = await _model.EvaluateAsync(new FriesOrNotFriesInput
            {
                data = ImageFeatureValue.CreateFromVideoFrame(VideoFrame.CreateWithSoftwareBitmap(await bitmapDecoder.GetSoftwareBitmapAsync())),
            });
            var label = output.classLabel.GetAsVectorView().FirstOrDefault();
            return Enum.Parse(label);
        }

        private async Task InitializeModelAsync()
        {
            if (_model != null)
            {
                return;
            }

            var onnx = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///Assets/FriesOrNotFries.onnx"));
            _model = await FriesOrNotFriesModel.CreateFromStreamAsync(onnx);
        }
    }
}

Add Android implementation

On Android platform, TensorFlow is popular library. In Java or Kotlin, there is tensorflow-android library.

TensorFlow AAR For Android Inference Library and Java API | Maven Repository

On Xamarin, there is wrapper library.

Xam.Android.Tensorflow | NuGet

The library was introduced following article of Xamarin Blog.

Using TensorFlow and Azure to Add Image Classification to Your Android Apps | Xamarin Blog

Add a model file and label file to Android project.

At next, I add the library to Android project, then create PhotoDetector.cs file to the project. At next, edit the file like below:

using Android.Graphics;
using Org.Tensorflow.Contrib.Android;
using Plugin.CurrentActivity;
using System;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Xamarin.Forms;

[assembly: Dependency(typeof(AIApp.Droid.PhotoDetector))]
namespace AIApp.Droid
{
    public class PhotoDetector : IPhotoDetector
    {
        private static readonly string ModelFile = "model.pb";
        private static readonly string LabelFile = "labels.txt";
        private static readonly string InputName = "Placeholder";
        private static readonly string OutputName = "loss";
        private static readonly int InputSize = 227;
        private readonly TensorFlowInferenceInterface _inferenceInterface;
        private readonly string[] _labels;

        public PhotoDetector()
        {
            _inferenceInterface = new TensorFlowInferenceInterface(CrossCurrentActivity.Current.Activity.Assets, ModelFile);
            using (var sr = new StreamReader(CrossCurrentActivity.Current.Activity.Assets.Open(LabelFile)))
            {
                _labels = sr.ReadToEnd().Split('n').Select(x => x.Trim()).Where(x => !string.IsNullOrEmpty(x)).ToArray();
            }
        }

        public async Task DetectAsync(Stream photo)
        {
            var bitmap = await BitmapFactory.DecodeStreamAsync(photo);
            var floatValues = GetBitmapPixels(bitmap);
            var outputs = new float[_labels.Length];
            _inferenceInterface.Feed(InputName, floatValues, 1, InputSize, InputSize, 3);
            _inferenceInterface.Run(new[] { OutputName });
            _inferenceInterface.Fetch(OutputName, outputs);
            var index = Array.IndexOf(outputs, outputs.Max());
            return (FriesOrNotFriesTag)Enum.Parse(typeof(FriesOrNotFriesTag), _labels[index]);
        }

        private async Task LoadByteArrayFromAssetsAsync(string name)
        {
            using (var s = CrossCurrentActivity.Current.Activity.Assets.Open(name))
            using (var ms = new MemoryStream())
            {
                await s.CopyToAsync(ms);
                ms.Seek(0, SeekOrigin.Begin);
                return ms.ToArray();
            }
        }

        private static float[] GetBitmapPixels(Bitmap bitmap)
        {
            var floatValues = new float[InputSize * InputSize * 3];
            using (var scaledBitmap = Bitmap.CreateScaledBitmap(bitmap, InputSize, InputSize, false))
            {
                using (var resizedBitmap = scaledBitmap.Copy(Bitmap.Config.Argb8888, false))
                {
                    var intValues = new int[InputSize * InputSize];
                    resizedBitmap.GetPixels(intValues, 0, resizedBitmap.Width, 0, 0, resizedBitmap.Width, resizedBitmap.Height);
                    for (int i = 0; i > 8) & 0xFF) - 117);
                        floatValues[i * 3 + 2] = (((val >> 16) & 0xFF) - 123);
                    }
                    resizedBitmap.Recycle();
                }
                scaledBitmap.Recycle();
            }

            return floatValues;
        }
    }
}

Add iOS implementation

The last platform is iOS. iOS has CoreML feature.

Core ML | Apple Developer Documentation

In Xamarin platform, you can use CoreML APIs. The documentation is below:

Introduction to CoreML in Xamarin.iOS | Microsoft Docs

Add the CoreML file to Resources folder of iOS project, and set CoreMLModel to Build Action.

At next, add PhotoDetector.cs to iOS project, then edit the file like below:

using CoreFoundation;
using CoreImage;
using CoreML;
using Foundation;
using System;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Vision;
using Xamarin.Forms;

[assembly: Dependency(typeof(AIApp.iOS.PhotoDetector))]
namespace AIApp.iOS
{
    public class PhotoDetector : IPhotoDetector
    {
        private readonly MLModel _mlModel;
        private readonly VNCoreMLModel _model;

        public PhotoDetector()
        {
            var assetPath = NSBundle.MainBundle.GetUrlForResource("FriesOrNotFries", "mlmodelc");
            _mlModel = MLModel.Create(assetPath, out var _);
            _model = VNCoreMLModel.FromMLModel(_mlModel, out var __);
        }

        public Task DetectAsync(Stream photo)
        {
            var taskCompletionSource = new TaskCompletionSource();
            void handleClassification(VNRequest request, NSError error)
            {
                var observations = request.GetResults();
                if (observations == null)
                {
                    taskCompletionSource.SetException(new Exception("Unexpected result type from VNCoreMLRequest"));
                    return;
                }

                if (!observations.Any())
                {
                    taskCompletionSource.SetResult(FriesOrNotFriesTag.None);
                    return;
                }

                var best = observations.First();
                taskCompletionSource.SetResult((FriesOrNotFriesTag)Enum.Parse(typeof(FriesOrNotFriesTag), best.Identifier));
            }

            using (var data = NSData.FromStream(photo))
            {
                var ciImage = new CIImage(data);
                var handler = new VNImageRequestHandler(ciImage, new VNImageOptions());
                DispatchQueue.DefaultGlobalQueue.DispatchAsync(() =>
                {
                    handler.Perform(new VNRequest[] { new VNCoreMLRequest(_model, handleClassification) }, out var _);
                });
            }

            return taskCompletionSource.Task;
        }
    }
}

How does it run?

This is results run on Windows 10.

On Android:

On iOS:

Conclusion

AI is very important technology. You can use it your apps on all platforms.
If you created apps using Xamarin, then you could add the AI feature by steps of this article.

Have a good programing.

Wer mit digitalen Medien im Schulalltag zu tun hat, kommt nicht um die Diskussion herum, ob der Fokus im Unterricht auf der Handschrift oder dem Tippen liegen sollte. Digitale Kompetenzen wie der Umgang mit der Tastatur sind auf dem zukünftigen Arbeitsmarkt von hoher Bedeutung, dort also, wo die heutigen Schülerinnen und Schüler morgen erfolgreich sein müssen. Die Handschrift fördert hingegen stärker sowohl die Motorik als auch kognitive Fähigkeiten und ermöglicht eine schnellere Entwicklung der Schreibkompetenz. Was ist also besser?

Schreiben mit der Hand aktiviert 12 Gehirnareale

Handschreiben übt in hohem Maße die Feinmotorik. Die schöne (oder weniger schöne) Handschrift ist zudem ein authentischer Ausdruck der Persönlichkeit. Aber dem nicht genug: Studien aus der Neurowissenschaft belegen, dass beim Schreiben mit der Hand Prozesse und Verknüpfungen im Gehirn entstehen, die beim Tippen auf einer Tastatur nicht vorhanden sind. Laut einer Metaanalyse aus dem Jahr 2013 werden beim Handschreiben zwölf Gehirnareale aktiviert. Auch das sogenannte Arbeitsgedächtnis funktioniert besser in Kombination mit der Handschrift. Hierbei werden Spuren im Gehirn abgelegt, die immer wieder abgerufen werden können, sodass die Kinder zum Beispiel später die gelernten Buchstaben schneller wiedererkennen. Das „Merken“ funktioniert einfach besser, wenn man mitschreibt und nicht etwa tippt.

Digitale Werkzeuge machen Lernen individuell

Der digital gestützte Unterricht hat sich an vielen Schulen bereits als sinnvoll und effektiv erwiesen und etabliert. So erreichen Schülerinnen und Schüler beispielsweise durch die Nutzung des plastischen Readers, der Teil der Microsoft Learning Tools und von Office 365 ist, erstaunliche Ergebnisse beim Lesen- und Schreibenlernen. Das liegt unter anderem daran, dass direktes Feedback sowie die Anpassung der Aufgaben an das individuelle Lernniveau jederzeit möglich sind. Alle Lernenden kommen also in ihrem eigenen Tempo voran.

Neben dem Spaß an neuen Technologien und einer damit einhergehenden Motivation lernen die Schülerinnen und Schüler kollaborativ und durch Konzepte wie beispielsweise den Skype-a-thon auch grenzüberschreitend und international. Das macht die Kinder und Jugendlichen fit für eine Zukunft, in der globales Denken, Zusammenarbeit und Empathie mehr denn je gefragt sein werden. Aber bedeutet die Anwendung neuer Technologien, dass in der Klasse nur noch getippt wird?

Das Beste aus zwei Welten

Die Nutzung neuer Medien und das Handschreiben stehen nicht im Widerspruch zueinander. Ganz im Gegenteil: In einer Unterrichtsstunde können sowohl der Schreibstift als auch das Tablet sinnvoll eingesetzt werden. Zahlreiche Best-Practice-Beispiele – oftmals aus den Niederlanden – zeigen, dass beides Hand in Hand gehen kann. Obwohl die Digitalisierung an niederländischen Schulen weitgehend fortgeschritten ist und so gut wie alle Klassen über Tablets verfügen, findet oftmals parallel der normale Schreibunterricht mit Übungsheft und Stift statt. Auch digitale Lösungen ermöglichen mittlerweile das handschriftliche Schreiben, etwa mit einem Tablet-Stift. Das kann sinnvoll sein, wenn das Geschriebene unmittelbar in Computerschrift umgewandelt werden soll, um zum Beispiel direkt in einem Dokument daran weiterzuarbeiten.

Und auch sonst hängt es immer von der Aufgabenstellung ab, ob besser ein digitales Werkzeug oder Papier und Stift zum Einsatz kommen. Sollen die Arbeitsergebnisse direkt überprüft werden? Soll auf verschiedenen Niveaus gelernt werden? Oder ist Buchlektüre nötig, bei der sich die Schülerinnen und Schüler Notizen zum besseren Einprägen machen sollen? Die heutige Schule kann beides haben – und damit das Beste aus zwei Welten.

Aufruf an Lehrkräfte: 15-minütige Online-Umfrage STEP 2019

Das Schreibmotorik Institut führt gemeinsam mit dem Verband Bildung und Erziehung eine bundesweite Online-Umfrage unter Lehrkräften durch. Die Daten sollen neue Erkenntnisse zum Handschrifterwerb und dem Einfluss digitaler Endgeräte sowie über die Dualität beider Methoden im Unterrichtseinsatz bringen. Nehmen Sie sich 15 Minuten Zeit und tragen Sie zu einem aussagekräftigen Ergebnis der Online-Befragung bei. Hier geht’s zur Umfrage.

When you view the properties of a volume in Explorer,
it reports (among other things) disk space used and disk space
remaining.
You find that you're running low on disk space,
so you want to look for big files that could be archived and deleted,
and you run a disk space tool,
but that tool reports disk usage less than what Explorer
reports.
Where is this secret disk space usage coming from?

Explorer obtains information about used and free disk space
by calling Get­Disk­Free­Space­Ex,
so the information comes straight from the file system.

The third party tool is probably calculating disk space used
by walking the entire disk and adding up file sizes.
And what's probably happening is that the secret disk space
usage is coming from directories that the third party tool
cannot access.
Since it cannot access them,
it just skips over them and assumes that the directories have
no files.

In particular, it probably skips over the
System Volume Information directory.
System Restore snapshots are kept there,
and they can get big.

Now the tricky part: How to clean up System Restore points?

Your first inclination might be to go to the Control Panel and
search for System Restore, and it will take you to the
System Protection control panel, where you can configure
System Restore.
But the only button available there is
Delete all restore points for this drive.
What if you want to clean up all but the most recent restore point?

To delete all but the most recent restore point,
you have to go to a completely different place:
The Disk Cleanup utility.

Anyway, from the Disk Cleanup utility,
click Clean up system files,
and then go to the More Options tab.
Under the
System Restore and Shadow Copies section,
click Clean up to clean up all but the most
recent restore point.

I don't know why there isn't a link from one to the other.

Bonus chatter:
Other sources of "mystery disk space usage" are
alternate data streams and general disk metadata,
like the directories themselves!

Over the last month, we’ve been excited to welcome thousands of new HockeyApp customers to Visual Studio App Center. As you may have read, last month we shared that HockeyApp will fully transition to App Center on November 16, 2019. Over this coming year, HockeyApp customers will discover all the benefits of App Center: the next generation of HockeyApp's distribution, crash reporting and analytics services, as well as powerful new services and features exclusive to App Center.
 
 

Get Started with your Transition Today

As a HockeyApp customer it’s easy to get started with your transition today. We've already migrated your HockeyApp apps to App Center, and are maintaining your existing data and experience in HockeyApp. We’ve built a side-by-side experience that allows you to work in both App Center and HockeyApp with your HockeyApp apps. Getting started is as simple as signing-in to App Center with your HockeyApp credentials.

Today, you can use the side-by-side experience to get familiar with App Center’s new interface and explore the new features and services. Starting early next year you’ll be able to complete your transition to App Center by moving your apps completely from HockeyApp to App Center. You can learn more about this cutover experience in another blog post recently published.

We’re Listening and We are With You Every Step of the Way

As you begin your transition, we’ll be with you every step of the way. We’ve created a Transition Center in the App Center dashboard with useful information and references to help you get started. Whether you have concerns, product feedback, feature requests or questions, our team is standing by in App Center. We're listening and would love to hear from you, so please don't hesitate to reach out.

We’ve built App Center with some great features so far, but this is only the beginning and we look forward to sharing additional information on our feature roadmap based on customer feedback soon. Please keep sharing your feedback and feature requests so we can continue building the future of App Center together. We’re excited about HockeyApp’s future in App Center and look forward to having you join us on this journey. See you in App Center!
 
 

The 13th cumulative update release for SQL Server 2017 RTM is now available for download at the Microsoft Downloads site. Please note that registration is no longer required to download Cumulative updates.
To learn more about the release or servicing model, please visit:

• CU13 KB Article: https://support.microsoft.com/en-us/help/4466404

Starting with SQL Server 2017, we are adopting a new modern servicing model. Please refer to our blog for more details on Modern Servicing Model for SQL Server

• Microsoft® SQL Server® 2017 RTM Latest Cumulative Update: https://www.microsoft.com/download/details.aspx?id=56128
• Update Center for Microsoft SQL Server: http://technet.microsoft.com/en-US/sqlserver/ff803383.aspx

Note: For this Cumulative Update release, it is intentional that SQL Server 2017 CU13 is releasing to Download Center on 12/18/2018 and releasing to Microsoft Update Catalog on 1/3/2019.  This is to abide with the Microsoft Update No Fly Zone during the end of year holidays.

Today we released PIX-1812.14, which adds support for selecting the playback adapter, displaying unbound resources and improvements to the occupancy lane.  This release also includes various miscellaneous fixes detailed below.

Playback Adapter Selection

PIX now allows you to select the GPU on which to play back a recording. After opening a wpix file, use this drop-down control in the top-right corner before starting analysis.

Note that workloads that explicitly use multiple adapters will be played back exclusively on the selected adapter, rather than on the default adapter as in previous versions of PIX.

Change to Capture API Selection Default

OS RS5 and up, the API drop-down on the “Launch Win32” and “Launch UWP” tabs allow you to control the set of DirectX APIs that PIX will monitor during capture.
In this release, PIX will now monitor DXGI and D3D11 activity by default. On RS5 and up, PIX is able to ignore DXGI and D3D11 activity altogether, and this was the default in the previous version of PIX. However, ignoring DXGI had a side-effect of exposing a lock-ordering issue between PIX and the DXGI runtime, resulting in a rare but possible hang when capturing with PIX. Unfortunately, DXGI and D3D11 have to be monitored together or not at all. Since PIX cannot capture D3D11 APIs, this new default means that PIX will fail to capture your application if you mix D3D11 and D3D12 rendering. We intend to fix these issues properly in a future release of PIX.

In the meantime, if your application uses D3D11 and D3D12, please use the API drop-down to select “ignore D3D11” (which also ignores DXGI), or “force 11on12” (which causes the OS to turn D3D11 API calls into equivalent D3D12 calls) as illustrated here:

A good example of a non-obvious mixed D3D12/D3D11 scenario is a XAML application that uses WinML.

Unbound Resources

PIX now displays entries for resources that are referenced by a shader but aren't actually bound to anything. If a title does this then, at best, it won't render what is wanted and, at worst, it'll TDR. Unbound resources are shown as red in the pipeline view:

Occupancy Improvements

When analyzing occupancy, PIX now displays a tooltip of per stage values and highlights areas with nonzero occupancy values. Improvements have also been made behind the scenes to speed up occupancy processing, particularly on GPUs with many compute blocks.

Known Issues

If you have both a NVIDIA Turing card (for example: a 2080) and a NVIDIA non-Turing card (e.g. a 960, 1080 etc) in the same PC, then Start Analysis will fail for GPU Captures. This can be worked around by going to Home -> Settings inside PIX, and unchecking 'Enable GPU Plugins'. We are working with NVIDIA to fix this issue in a future PIX release.

Other Changes

GPU Captures

• Support for Render Passes
• Show warning dialog if developer mode is not recognized by d3d
• Fix display of structured buffers
• Fix buffer formatting when same resource bound to multiple stages
• Improve display of "not set" constant values
• Hide CUDA-only counters in PIX
• GPU captures no longer run dxdiag (and so don't have to wait for it to complete)
• Fix crash when opening captures that don't contain a dxdiag report
• Fix bug where PIX doesn't correctly capture an app which has D3D12.dll in the same directory as the exe
• Fix GPU capture timing data for metacommands
• Add clear error message is application builds/copies acceleration structure into invalid ID3D12Resource
• Various shader table improvements

Timing Captures

• Make timing data's SDK layer warning be more accurate

Misc

• Improve call stack output
• Fix issue on systems with low virtual memory sizes
• Fixed minor memory leak involving remote connections

With Azure IoT Edge 1.0.5 release, developers should use process isolation container to build and run IoT Edge modules for Windows RS5 IoT Edge device. And Azure IoT Edge tools for Visual Studio and Azure IoT Edge for Visual Studio Code also updated its default project template to align with RS5 requirements. You can follow the tutorials for VS or VS Code to build RS5 C# module with the dockerfile.windows-amd64 in the module folder. However, if you still want to use Hyper-V container for Windows RS4 (or before) IoT Edge device. Here comes the instructions.

Continue developing IoT Edge RS4 module in Visual Studio Code

Before started, making sure you have installed 1.7.0 (or after) version of Azure IoT Edge for Visual Studio Code. Follow the instructions below to stick with IoT Edge 1.0.4 and RS4 module templates.

1. Open a new VS Code window, in Commend Palette, type and open Preferences: Open Settings.
2. Navigate to Extensions -> Azure IoT Edge configuration.
   
3. Make sure you have checked Template Install for all the options. By doing this, VS Code will automatically install your specified version of the module template.
   
4. Specify the module template version so that you can stick to RS4 module project template. You can update the value for all the templates or just update the template you want to use.
   1. C# function module template - 3.0.1
   2. C# custom module template - 2.3.0
   3. Java custom module template - 1.2.0
   4. Node custom module template, run npm i -g generator-azure-iot-edge-module@1.2.0 in the terminal
5. Make sure your Edge Agent and Edge Hub versions are 1.0.4.
   

After doing that, each time when creating a new solution or adding a new module for RS4, you will get 1.0.4 for your EdgeHub and EdgeAgent in your .template.json and RS4 base image the dockerfile.windows-amd64.

Once you decide to move to RS5, just delete the version numbers you've specified in the settings and create new solution or add a new module.

Continue developing IoT Edge RS4 module in Visual Studio 2017

Azure IoT Edge tools for Visual Studio (Preview) has updated the C# template to use RS5 images. If you still want to develop and deploy RS4 C# module. Follow the instructions below to handle the version number and base image manually.

1. Open Dockerfile.windows-amd64, update the base image from microsoft/dotnet:2.1-runtime-nanoserver-1809 to microsoft/dotnet:2.1-runtime-nanoserver-1803.
   
   
2. Open deployment.template.json file, update the version number for EdgeHub and EdgeAgent from 1.0 to 1.0.4.
   

Once to decide to move to RS5, use 1809 base image and 1.0 for your Edge runtime in your existing projects. All newly created projects will use RS5 template by default.

I have moved my blog to Dynamics 365 Community.

The School of Computer Science (SoCS) at Lincoln has seen rapid growth in both student and staff numbers in recent years, recently appointing four new Professors to extend its research focus and base. Domiciled within the College of Science, the School sits in the top 20% nationally for NSS overall satisfaction for the subject area of Computing. Benefitting from state-of-the-art facilities, the School offers a broad range of expertise in computing technologies and information systems, including specialisms in robotics and autonomous systems; computer vision and image engineering; medical applications of technology, and social and games computing. In addition to this wide range of expertise, our students have access to free Microsoft Technology Associate (MTA) certification which enhances employability and career pathways, and compliments their academic programmes.

Industry 4.0 Curriculum at Lincoln

SoCS is focused on the development of cutting edge Industry 4.0 (ID 4.0) curriculum, a response to the UK government’s ‘Digital Skills Crisis’ report [1], and digital industrial strategy that encompasses widening the national ID 4.0 agenda [2]. This approach supports the SoCS strategy of equipping our graduates with next generation ID 4.0 skillsets that are currently in short supply in the Computing industry. SoCS staff are steering committee members of a HEFCE funded Industrial Digitalisation project with the primary focus of embedding ID 4.0 content in programmes across the College of Science such as Engineering, Computing, and Maths/Physics [3]. An additional part of the project’s remit is to explore embedding cross-disciplinary ID 4.0 content across non-STEM subject areas.

Isaac Newton Building, home of the Lincoln School of Computer Science

MTA Certification Strategy

To supplement the school’s ID 4.0 curriculum, further value has been integrated in all undergraduate programmes of study by giving students the opportunity to sit an MTA exam for each year for their programme. Essentially, this means a SoCS graduate could achieve three distinct MTA certifications to take into industry in their chosen career pathway. The school currently has around 700 students enrolled on undergraduate programmes across Computer Science and Games Computing, as such the school has built the capability to manage MTA certification at scale with fantastic results. The benefits of Industry Certification to supplement academic programmes are clear, leading to improved career opportunities and a unique selling point for graduates when entering the employment market. The school has already identified instances were recent graduates have gained desirable employment as a direct result of attaining certification, which is excellent news to support validating the effectiveness of certification.

At present the school offers the full MTA certification portfolio to students through a targeted strategy. For students in their first year, the emphasis is on achieving certification across the programming MTA offerings, in second year the emphasis is on Networking and Databases certification, with the final year opening up all MTA certifications to students. This approach for final year students facilitates the benefit of being able to choose an MTA that best aligns with their career aspirations. Exam sits are planned alongside the university-wide exam weeks in January and May for first and second years students. For final year students it is acknowledged this is the most challenging year of their academic programme, and therefore more flexibility is desirable on when they can sit MTA exams. This is done through offering multiple exam sits throughout the academic year that are mindful of the school’s overall assessment submission strategy.

Student Results

The school completed its first rollout of MTA certification between November 2017 – May 2018. During this time the initiative experienced very high levels of student engagement and received positive feedback. In terms of learning resources for revision, student has free access to the Gmetrix service, supplementing the learning resources available from the school’s Microsoft Imagine Academy membership. Students reported they preferred a wide range and type of learning resources available to them to revise for MTA exams, indeed this caters for preferences across different resources such as video tutorials, textbooks, and mock exams.

Our statistics show that between November 2017 and May 2018 SoCS students sat 378 MTA exams with a pass rate of 73%, see table below for further details. The school is delighted by such high levels of engagement and achievement, particularly when MTA exams are offered to students as an extra-curricular activity on top of their academic programmes. As a result of this success the school is committed to running MTA exams for current and future students. In the current academic year, MTA exams are scheduled for January 2019 with 250 students currently signed up, with multiple exam sit opportunities planned between February-May 2019. It is envisioned that MTA exams will continue to be offered to SoCS students in the longer term, and we are keen to work with Microsoft to help develop MTA exams and share best practice as we continuously evolve and reflect on enhancing the student experience at Lincoln.

Number of MTA exam sits @ University of Lincoln between November 2017 – May 2018

Graduate Success with MTA

[1] https://publications.parliament.uk/pa/cm201617/cmselect/cmsctech/270/270.pdf

[2] https://beisgovuk.citizenspace.com/strategy/industrial-strategy/supporting_documents/buildingourindustrialstrategygreenpaper.pdf

[3]https://webarchive.nationalarchives.gov.uk/20180319121039/http://www.hefce.ac.uk/funding/catalyst/futureskills/

Deploy cheaper low priority (preemptable) instances of Ubuntu Data Science Virtual Machine ‘DSVM’ using the Microsoft Azure portal

Deploy to Azure via Github

-We have a set of deploy to Azure Scripts available to deploy the low priority VM simply click the link below

There are additional scripts here on the DSVM GitHub repo for deploying Linux or Windows versions

For example: NC6 (1 K80 GPU) low-priority (preemptible) instance costs $0.18/hour compared to $0.90/hour for a dedicated instance (depending on region).

Please note this template is setup to delete the disks upon eviction. So you want to save any data on the VM externally if you need to, on a blob or Azure files. You can update the template to set eviction policy to “deallocate” to keep the disks (storage costs apply). 

Low Priority Pricing

See low priority instance pricing for the latest price for the specific region

Adding Custom Features

The template can be found here on the DSVM github repo if you want to customize it.

Low-priority VMs’ Anatomy

Microsoft grants access to their spare capacity in each region. The amount of spare capacity varies by region and VM size according to multiple factors, including day of week, time of day, and demand for different VM sizes.

With low-priority VMs, Microsoft effectively lets you take advantage of unused capacity for a great price. The data center team can reclaim these VMs at any given moment due to higher priority allocations so the capacity isnt fixed.

Using Batch Pools

Currently, Low-priority VMs can be easily purchased via Azure Batch, a service that lets you run jobs and tasks across multiple compute pools, “Batch pools”.

Each Batch pool can contain both on-demand and low-priority nodes which can be independently scaled.

Pricing

Low-priority Linux VMs come with 80% discount while the Windows VMs come with 60% discount. The discount is calculated in comparison to their On-Demand hourly price. The vast majority of VM sizes and types are supported to run as Low-priority VMs.

So we have seen many academics keen to use Low-priority VMs to run and scale their course or research infrastructure cost-effectively.

Here are some examples of common use cases that are suitable for Low-priority VMs.

• Web Services
• Testing
• CI/CD Pipelines
• Containers Platforms (Kubernetes, Mesos, Docker Swarm, etc..)
• Batch jobs such as Media processing, Transcoding, and Rendering.
• Batch jobs such as Machine Learning Data Science.

We look forward to helping you to get started and consuming Low-priority VMs on Microsoft Azure and achieve business agility, cost savings, and scale.

Using Azure Managed Service Identity (MSI) to provide students access to Azure Resources

To learn how to use managed identities to access different Azure resources, try these tutorials.

Learn how to use a managed identity with a Windows VM:

• Access Azure Data Lake Store
• Access Azure Resource Manager
• Access Azure SQL
• Access Azure Storage by using an access key
• Access Azure Storage by using shared access signatures
• Access a non-Azure AD resource with Azure Key Vault

Learn how to use a managed identity with a Linux VM:

• Access Azure Data Lake Store
• Access Azure Resource Manager
• Access Azure Storage by using an access key
• Access Azure Storage by using shared access signatures
• Access a non-Azure AD resource with Azure Key Vault

Learn how to use a managed identity with other Azure services:

• Azure App Service
• Azure Functions
• Azure Logic Apps
• Azure Service Bus
• Azure Event Hubs
• Azure API Management
• Azure Container Instances

Using MSI for providing student access to Azure ML Workspace from a dedicated Data Science Virtual Machines

Typically courses provide a fixed pool of resources for students to utilise.

Pooled Resources

Many academics are providing Azure Data Science VM (DSVM) for the class.

Using Managed Services Identity

When you create the VM with a MSI and give it permission to the resource group where you have the Azure ML Workspace. In this scenario any local user on the DSVM (students in the lab) can access the Azure ML Workspace without having a dedicated Azure subscription credentials.

The Students simply need to authenticate to the Virtual Machine (using local auth/AD/Shibboleth). 

Below is the sample code to do add Managed Service Identity to create or attach to a Azure ML Workspace with the change from usual highlighted.

Please note, MSI is a general Azure feature and not specific to DSVM as can be seen in the examples above so you can use MSI to provide students access to centrally provided resources..

Ref: https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview

Overview

SQL Server relies on Forced-Unit-Access (Fua) I/O subsystem capabilities to provide data durability, detailed in the following documents: SQL Server 2000 I/O Basic and SQL Server I/O Basics, Chapter 2

Durability: “In database systems, durability is the ACID property which guarantees transactions that have committed will survive permanently. For example, if a flight booking reports that a seat has successfully been booked, then the seat will remain booked even if the system crashes.” - https://en.wikipedia.org/wiki/Durability_(database_systems)

Durability is a cornerstone of any database system and starting with SQL Server 2017 on Linux Cumulative Update 6 (CU6), SQL Server on Linux enables “Forced Flush” behavior as described in this article, improving durability on non-Fua optimized systems.

“Be sure to deploy SQL Server 2017 CU6 or newer for best data durability and performance results.“

Microsoft SQL Server Database Engine Input/Output Requirements

The Back Story

My involvement began when looking at the Write-Ahead-Logging (WAL) and the write behaviors on a Linux SQL Server. The investigation revealed that some of the Linux file systems may not be Fua optimized. Instead of passing the Fua bit with the write request (like Windows) Linux generates a separate device level flush request. Device level flushing may have an impact on your I/O caching, read ahead or other behaviors of the storage system. The “forced flush” changes in SQL Server avoid flushes, when possible, in order to improve performance on non-optimized Fua file systems.

Performance points of reference: The following charts highlight the increase in performance.

4 Socket, TPCC 9,108 – Old kernel 36,618 – Patched kernel

2 Socket, TPCC 9,606 – Old kernel 18,840 – Patched kernel

During my investigation, I traced, tested, reviewed, and instrumented the Linux kernel to understand the SQL Server I/O durability needs and associated impacts. This effort resulted in:

· Changes to the SQL Server Engine
· Changes to the SQL Server Linux Host Extension
· Validation with the Windows Storage team
· Validation with Windows Hyper-V team
· Validation with Windows Azure Storage team
· Conversations with Red Hat, HP, SUSE and Intel
· Changes in the Linux file system

Before all was said and done, all of these people and more helped with various activities:

· Eugene, Mike, Slava, Scott, Peter, Steve and others doing code reviews on my SQL Server and Host Extension changes.
· Val and Dylan performing power outage durability testing and adding new tests.
· Mitch and Ben doing functionality testing with the matrix of trace flags, mssql.conf options, Red Hat, Centos, Ubuntu, SUSE, and Windows using xfs, xfs ^opt, ext4, refs and ntfs.
· Pat validating Docker functionality and updating SQLIOSim to allow forced flush patterns.
· Patrick and Purvi doing performance and regression analytics.
· Kapil building and validating behavior on the Linux kernel changes for xfs (Note: I have tagged it xfs ^opt (xfs optimizations) in this document for reference.)
· Amit, Lee, Vaqar and Mike handling release activities and documentation.
· Neal, Matt, and others from Windows Storage, Windows Azure Storage, Windows Hyper-V, … validating Windows behaviors.
· Red Hat, SUSE, HPE, Intel helping and confirming Linux behaviors and making file system code changes.
· Pradeep, Venu and Suresh handling support issues. 

Special thanks to Dave Chinner and the Red Hat team, Dan Williams, Hamesh Patel and their colleagues from Intel, Scott Norton and his colleagues, Josh Paulson and members of the Microsoft Linux teams, Jan Kara from SUSE and many others.

Terms

Before diving into details the terminology is important.

Term	Type	Description
Stable Media	Concept	Any storage device that can survive a power outage.  In the 90’s we talked about saving data on spinning media.  SSDs, persistent memory, battery-backed controllers broaden the range of devices surviving power outage from traditional, spinning media.
WAL	Concept	Write-Ahead Logging or journaling.  The log records, which can be used to recreate the data, are stored in stable media before the data buffers are written and data buffers are stored in stable media before the supporting log records can be deleted.
Fua	Hardware	Fua = Forced Unit Access.  Fua is the standard used to issue an I/O write command in stable data storage.  Fua has been a long-standing part of the SCSI specifications.  Caution: Non-SCSI storage implementations often silently ignore the request.
Device Flush	Hardware	A device level flush forces any dirty buffer located in volatile storage onto stable media.  Flushing a non-battery backed disk or cache device can be performance impacting and repeated device flush commands reduce the optimized capabilities, such as, writes near current head placement, wear level reduction, caching mechanisms, etc.
FILE_FLAG_NO_BUFFERING	Windows	FILE_FLAG_NO_BUFFERING is the Win32, CreateFile API flags and attributes setting to bypass file system cache.  This flag does NOT imply flush behaviors, only the avoidance of the system file cache.  FILE_FLAG_NO_BUFFERING maps to the O_DIRECT flag for the Linux open command.   Warning: The flag causes the file system cache of a guest VM to be bypassed but the VM or container may not pass the option to the host.
O_DIRECT	Linux	Linux open command flag used to bypass file system cache.   Warning: The flag causes the file system cache of a guest VM to be bypassed but the VM or container may not pass the option to the host.
FILE_FLAG_WRITE_THROUGH	Windows	FILE_FLAG_WRITE_THROUGH is the Win32, CreateFile API flags and attributes setting generating Forced Unit Access (Fua) writes.  A write honoring Fua is issued with the Fua flag and does not return from the write command until transferred to stable media.    FILE_FLAG_WRITE_THROUGH maps to the O_DSYNC flag option for the Linux open command.   The FILE_FLAG_WRITE_THROUGH flag does NOT imply FILE_FLAG_NO_BUFFERING behavior. To achieve no buffering and Fua, both the FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH flags must be specified.
O_DSYNC | fdatasync | RWF_ODSYNC	Linux	The O_DSYNC | RWF_ODSYNC | fdatasync indicate to the Linux file systems that the ‘data integrity standard’ is to be upheld for I/O operations.  The data integrity standard makes sure the data and supporting meta data, needed to retrieve the data, are stored in stable media.   O_SYNC and fsync indicate to the Linux file system to uphold the ‘file integrity standard.’  O_SYNC is a superset of the O_DSYNC as the standard indicates that all file metadata (access timestamps for example) are stored in stable media.   RWF_ODSYNC is a newer variation of O_DSYNC.  The RWF_ODSYNC can be set as a flag for each I/O request.  Whereas, opening the file with O_DSYNC establishes flushing behavior for all writes to the file.  SQL Server does NOT make use of RWF_ODSYNC.
FILE_FLAG_OVERLAPPED	Windows	FILE_FLAG_OVERLAPPED is the Win32, CreateFile API flags and attributes setting enabling asynchronous API behavior.  Control from APIs such as WriteFile and ReadFile returns as soon as the I/O is queued.  The operating system signals completion when the I/O stack finishes the request.
AIO | io_submit | Kernel I/O (kio)	Linux	Linux provides different system calls (syscall)/commands for asynchronous behavior.   ·         write command – synchronous syscall, does not return until the write completes. ·         aio_write | io_submit – asynchronous syscall, returning control as soon as the I/O request is queued. SQL Server uses aio* for asynchronous I/O operations.
Synchronized	Concept	Not to be confused with the synchronous behavior of the API/ABI invocations.  In the context of I/O, synchronized indicates that the storage and in-memory copies of the data are the same.  O_DSYNC often uses this term to indicate that the in-memory buffer is in sync with the storage media.
Data Integrity Standard	Concept	This data integrity standard is a subset of the file integrity standard.  In summary, the data integrity standard is the ability of the file system to store and retrieve data over a power outage.  This includes the data itself as well as the tracking structures for the storage.   This diagram shows a high-level view of a File.  The file holds the metadata, made up of attributes (time of last access, permissions, …) as well as mappings to actual storage (sectors on a disk, pages on an SSD, …)   Flush to stable media occurs for the data and storage mappings to achieve the data integrity standard.   Refer to the ISO documentation on data integrity for complete details. CSA ISO/IEC 9945-1:2005 (R2009) 3.375 Synchronized I/O Data Integrity Completion
File Integrity Standard	Concept	This file integrity standard is a superset of the data integrity standard.  The file integrity standard is the ability for the file system to store and retrieve data and metadata written over a power outage including attributes such as the time of last access and permissions.   Refer to the ISO documentation on file integrity for complete details: CSA ISO/IEC 9945-1:2005 (R2009) 3.376 Synchronized I/O File Integrity Completion
O_NONBLOCK	Linux	Linux open command flag value to avoid blocking during an I/O call.  This is not a FILE_FLAG_OVERLAPPED implementation for the Linux ABIs.  Instead, the flag is used to decide if a wait will occur while attempting to issue the I/O request.  If the thread would become blocked the attempt to issue the request is aborted and control is returned to the caller.   Overlapped will issue the I/O and return STATUS_PENDING, whereas the O_NONBLOCK returns EAGAIN and the caller must retry the entire I/O operation.
O_NOATIME	Linux	Linux open command flag to avoid updating the time of last access which can reduce metadata writes.  The O_NOATIME flag does NOT disable MTIME or CTIME modifications.
REQ_FUA	Linux	Block device write request using the Fua capabilities when performing a write.  The Fua bit is sent with the write signaling stable media storage is required.
REQ_PREFLUSH REQ_POSTFLUSH	Linux	A block device write request (0 bytes – write[10, 12 or 16 bytes] SCSI command) requesting a device flush.
DpoFua	Linux	DPO = Disable Page Out (for writes instructs I/O subsystem to avoid adding write data to the cache, for reads prevents replacing existing entries in cache.)   Fua and Dpo are separate bit values in the SCSI SCB.  SQL Server only requires Fua and not DPO capabilities.
Forced Flush Behavior	SQL Server	When enabled the Linux SQL Server opens the file with FILE_FLAG_NO_BUFFERING and without FILE_FLAG_WRITE_THROUGH (no FUA.)  SQL Server log and checkpoint made durable with calls to FlushFileBuffers.

When SQL Server performs a write, using a file opened with FILE_FLAG_WRITE_THROUGH(O_DSYNC), Linux performs the multi-step operation as a data write followed by the complete device flush request instead of adding the Fua bit to the data write request.

1. Issue write to block device for the data
2. Receive the data write complete event in the block device’s write completion routine
3. If O_DSYNC requested, issue block device flush
4. Wait for block device flush request completion
5. Return control to write caller

Every write, requiring O_DSYNC behavior, also invokes a block device flush on Linux.  The repeated use of a write request followed by a flush request may be detrimental to performance and will increase traffic on the I/O bus.

Note: The 4.18 Linux Kernel updates allow the XFS file system and a device supporting Fua writes to avoid the additional block device flush request.

Prior to the Linux Kernel 4.18 updates, Linux could use Fua but only for the file system journaling writes and not data writes.  Linux queries the device for the DpoFua support capabilities and when the device reports Fua support (DpoFua = 1) the journaling is optimized to use Fua writes.  The data storage always uses the multi-step flush process. 

Windows does NOT query the device for support, instead any file opened with FILE_FLAG_WRITE_THROUGH sends the Fua option with the write request where the target device is expected to honor the Fua option.  In some instances, the target silently ignores the request. The Windows implementation places the responsibility of making sure the I/O subsystem is Fua compliant on the database and system administrator.

Fua (Forced Unit Access)

Microsoft SQL Server requires Fua compliant storage (stable media) ensuring the data is stored in stable media before a write request returns to SQL Server.  For example, if you run SQL Server on an IDE or SATA drive, you may need to disable the physical disk cache to accommodate WAL (durability), requirements. The SQL Server 2017 CU6, forced flush feature, provides durability assistance on Linux systems without Fua.

Note:  Disabling the write cache using the Windows, properties dialog may not suffice.  Some drives are not compliant with the dialog options and require special utilities to configure the drive cache.  Linux may need similar utilities to control various hardware cache installations.

You can explore your system using the df and lsblk commands.  When Linux mounts a block device, Linux queries the attributes and capabilities of the block device.  One such capability is the ability to support DpoFua for write requests.  When DpoFua=1 is reported, the device is reporting support for Fua.

Note: These commands may require root permissions to execute.

df -T

Filesystem          Type     1K-blocks     Used Available Use% Mounted on

devtmpfs            devtmpfs   4062840        0   4062840   0% /dev

tmpfs               tmpfs      4078624        0   4078624   0% /dev/shm

tmpfs               tmpfs      4078624     8880   4069744   1% /run

tmpfs               tmpfs      4078624        0   4078624   0% /sys/fs/cgroup

/dev/mapper/cl-root xfs       52403200 13010400  39392800  25% /

/dev/sda1           xfs        1038336   431168    607168  42% /boot

/dev/mapper/cl-home xfs      114361796 22488816  91872980  20% /home

tmpfs               tmpfs       815728        0    815728   0% /run/user/1000

lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

fd0           2:0    1     4K  0 disk

sda           8:0    0   168G  0 disk

├─sda1        8:1    0     1G  0 part /boot

└─sda2        8:2    0   167G  0 part

  ├─cl-root 253:0    0    50G  0 lvm  /

  ├─cl-swap 253:1    0   7.9G  0 lvm  [SWAP]

  └─cl-home 253:2    0 109.1G  0 lvm  /home

sdb           8:16   0     1G  0 disk

sr0          11:0    1  1024M  0 rom

The /dev/sda1 entry reports an xfs file system and the device id of 8:1 (major:minor.)  The block device is sda (8:0.)  The entries for sda1, sda2 are partitions on the same block device.

You may also use lshw to explore disk information such as sector size and vendor which are helpful in determining additional device details.

lshw -class disk

       description: SCSI Disk

       product: Virtual Disk

       vendor: Msft

       physical id: 0.0.0

       bus info: scsi@0:0.0.0

       logical name: /dev/sda

       version: 1.0

       size: 168GiB (180GB)

       capabilities: partitioned partitioned:dos

       configuration: ansiversion=5 logicalsectorsize=512 sectorsize=4096 signature=000a0600

Use dmsg or sg_modes to reveal the DpoFua state.

dmesg | grep sda

[    1.266561] sd 0:0:0:0: [sda] 352321536 512-byte logical blocks: (180 GB/168 GiB)

[    1.266562] sd 0:0:0:0: [sda] 4096-byte physical blocks

[    1.266892] sd 0:0:0:0: [sda] Write Protect is off

[    1.266894] sd 0:0:0:0: [sda] Mode Sense: 0f 00 10 00

[    1.266949] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and Fua

[    1.271227]  sda: sda1 sda2

[    1.274579] sd 0:0:0:0: [sda] Attached SCSI disk

[    4.637845] XFS (sda1): Mounting V5 Filesystem

[    4.661270] XFS (sda1): Ending clean mount

Note:  /dev/sda is the block device and /dev/sda1, /dev/sda2 are partitions of the block device.

sg_modes /dev/sda1

    Msft      Virtual Disk      1.0    peripheral_type: disk [0x0]

Mode parameter header from MODE SENSE(10):

  Mode data length=20, medium type=0x00, WP=0, DpoFua=1, longlba=0

  Block descriptor length=0

>> Caching, page_control: current

00     08 0a 04 00 00 00 00 00  00 00 00 00

If your system supports Fua and you have the Linux Kernel 4.18 or newer updates, you can enable SQL Server trace flag -T3979 and use /opt/mssql/bin/mssql-conf set control.alternatewritethrough 0.  SQL Server will use Fua write behavior patterns instead of Forced Flush.  File systems supporting optimized Fua align SQL Server on Linux with SQL Server on Windows behavior and performance.

Note: If DpoFua=0 is reported the Linux file systems revert to the write, flush behavior to support O_DSYNC.  If you are unsure about the Fua support use forced flush behavior.

 

Note: DpoFua optimizations may be ignored during the first data write to an offset in the file, as documented here.

 

Note: Some systems embellish the Fua support capabilities.  You should always validate the Fua capabilities with your device and hardware vendor.

Beware The Cache

The following is a high-level flowchart of the settings and outcomes.

Safety and Performance

To help make sense of all of this I have broken down the various trace flag and configuration options by data safety and performance. 

Safe Installation (Best performance):
- SQL Server 2017 CU6 or newer
- Linux kernel 4.18 or newer with xfs storage
- T3979
- DpuFua=1
- control.writethrough = 1
- control.alternatewritethrough = 0

Safe Installation (Best Grouping Performance – Forced Flush from SQL Server and HE):
- SQL Server 2017 CU6 or newer
- T3982
- DpuFua = 0
- control.writethrough = 1
- control.alternatewritethrough =1

Safe Installation (Midrange Grouping Performance - Grouped device level flushes from HE):
- SQL Server 2017 CU6 or newer
- T3979
- DpuFua = 0
- control.writethrough = 1
- control.alternatewritethrough =1

Safe Installation (Slowest Performance - Linux kernel write, flush O_DSYNC behavior):
- SQL Server 2017 CU6 or newer
- T3979
- DpuFua = 0
- control.writethrough = 1
- control.alternatewritethrough =0

Alternative Installation (O_DIRECT Safe Reliance on I/O Subsystem):
- SQL Server 2017 CU6 or newer
- T3979
- DpuFua = 0
- control.writethrough = 0
- control.alternatewritethrough =0

      * Note: This achieves the prior configuration of SQL Server 2017 RTM to CU5 on Linux using only O_DIRECT are relying on O_DIRECT safe installations.

O_DIRECT safe refers to is the ability to perform writes with O_DIRECT and without O_DSYNC and maintain durability.  If the I/O subsystem avoids the proper buffering mechanism until it reaches stable media (battery-backed cache for example) then the Fua option may not be required.  However, to be O_DIRECT safe you must ensure the kernel, drivers, and hardware are able to place a write into non-volatile storage before indicating the write is complete.

I am unaware of tracing mechanisms to determine if the system is O_DIRECT safe.  Microsoft has done extensive testing of durability on systems believed to be O_DIRECT safe and SQL Server for Linux RTM thru CU5 was configured for O_DIRECT safe write activities.  However, Microsoft does not recommend reliance on O_DIRECT safe configurations and instead provides SQL Server 2017 CU6 and newer builds with explicit durability capabilities (forced flush or optimized Fua writes.)

If you choose to configure your system for O_DIRECT safe writes be sure to employ an appropriate backup or high availability strategy.

Caution

Unless you are using an updated file system, it is unsafe to assume the REQ_FUA behavior.  SQL Server automatically enables -T3982 (forced flush behavior) on SQL Server for Linux.  If you are able to, verify that a write on your system meets the following criteria, you may be able to achieve data integrity compliance without Fua capabilities as long as you are able to make sure the O_DIRECT path supports write completion when the data has been stored on stable media. – “O_DIRECT Safe.”

 

1.     I/O subsystem and drivers do not cache FILE_FLAG_NO_BUFFERING (O_DIRECT) in volatile cache. All cache involvement must be power outage hardened.

2.     All files in the database are on these stable media, compliant I/O paths (data, log, checkpoint files, standby recovery files, backups, …)

3.     All SQL Server support files are on the stable media (cluster files, registry files, …)

 

If you are unsure maintain the forced flush behavior configuration.

Virtual Environments (VMs and Containers)

Virtual environments may not provide the write behavior required to maintain durability.  Microsoft has validated Hyper-V, Windows Azure Virtual Machines, Windows Storage and various Docker implementations pass the flush requirements to the host properly.  When performing a Fua write the Fua requirement must be passed to the host and the host must carry the write in stable media.  If the virtualization or host caching does not support the Fua requirement you should use forced flush behaviors to maintain durability.

How did I get DpoFua=1 enabled for a Hyper-V installation of Linux?

On virtual systems, additional effort may be required to confirm the virtualization technology passes the Fua intent to the host and the host honors the request.  You may use a directly mounted device (direct pass-through drive for Hyper-V or iSCSI supporting Fua capabilities) and/or confirm the VHD | VHDx storage path supports Fua. 

WARNING: Use caution as the Hyper-V configuration does not query the device for Fua support and trusts the setting provided.

Using the Hyper-V, PowerShell cmdlets, from Windows 2016, you can configure Fua reporting for the guest virtual machine.

$a = Get-VMHardDiskDrive -VMName "xxxxxxxxxxx"
foreach ($b in $a) { $b; $b.WriteHardeningMethod }

Set-VMHardDiskDrive -VMName "xxxxxxxxxxx"  -OverrideCacheAttributes WriteCacheAndFuaEnabled

     Note: You may need to add a controller type such as -ControllerType SCSI

Linux Kernel 4.18 Updates

I have worked at Microsoft since 1994 and this issue truly points to the spirit of ‘Microsoft Loves Linux.’  I was strongly supported, funded and encouraged to track down the issue and make it better for customers.  The Linux committer and vendor engagements where insightful and helpful and, in the end, I am able to proudly say that Microsoft helped to improve Linux.

Several code changes were made to the imap, block device, and XFS components to accommodate optimized Fua writes.  The changes in the imap and block device areas can be leveraged by other file systems but as Dec 2018 (this post) only XFS has enabled optimized Fua data write capabilities.

Key Commit References in 4.18 and newer kernel releases

• ed5c3e6 xfs: move generic_write_sync calls inwards
• 4f8ff44 iomap: iomap_dio_rw() handles all sync writes
• 0ce9144 block: add blk_queue_fua() helper function
• 3460cac iomap: Use FUA for pure data O_DSYNC DIO writes

Support
Microsoft has tested and confirmed the Fua changes are included in Linux kernel build 4.18 and newer kernels while Red Hat has tested and confirmed the changes are included in RHEL 8.0.

Hint: Use the uname -sr command to discover the Linux Kernel version.

Tracing SQL Server Behavior (XEvent)

The following XEvents can be used to monitor the flush behavior employed by SQL Server.

XEvents (Debug channel)

·         flush*

·         make_durable*

·         checkpoint*

-T3979 (Force Flush Disabled) Checkpoint

-T3982 (Force Flush Enabled) Checkpoint

Notice the make_writes_durable and flush_file_buffers events for the database log and data files used for ACI(D)urability.

Testing and Futures (Warning: Plans May Change)

When it comes to database consistency there is no discussion on doing the right thing.  When I told Slava I was going to put in the forced flush behavior and it would have a performance impact, he simply said do it!  Not 2 seconds passed between us making the decision. As a team, we would figure out ways to improve the performance, but the safety of the data took priority.

When SQL Server 7.0 shipped Microsoft SQL Server made a distinct change to transaction log writes, aligning on sector boundaries and removing the 2K page rewrites of the 6.x and Sybase code to improve ACID properties of the SQL Server.  This generated lots of calls to Microsoft Support because it changed the I/O performance characteristics in ways customers didn’t expect and lead to the first code I added to SQL70IOStress, which I would later rewrite into what you know today as SQLIOSim. 

I mention this because folks like Keith Elmore, Bob Ward and I were running tests in the lab.  In fact, I was pulling the power between the power supply and the hard drive, waiting a few minutes, plugging the power back in and validating the durability and consistency provided by the I/O subsystem.  This lasted a couple of days until I electric-arc’ed the drive and burnt out the power supply. The lab manager would no longer allow me to manually pull the power.  It just so happened that Ed Harper was on our team and he built me a wire wrapped board.  It had a few D-Batteries and a switch on it with a plug on one end for the power supply and a plug on the other for the drive.  I used the generic driver sample and could signal the switch to open and close which I could then signal from SQL70IOStress and perform 1000s of ACID related tests overnight.  I won’t even tell you how many disk drives I have worn out doing I/O testing.

Creating SQL 2017 CU6 updates involved revisiting similar tests.  The SQL Server test infrastructure can inject fail points.  Dylan added additional fail points to checkpoint, lazy writer, and log writer.  Ben, Mitch, and Kapil performed physical pull the plug and kill the VM testing.  Pat added forced flush behavior to SQLIOSim, which we have been running in a Linux package, in our daily lab runs since before SQL Server 2017 was released.  We also added additional coverage to containers, high-availability, replication, backup, restore and recovery test suites.

Met Tejas, a long time Microsoft SQL Server Support Escalation Engineer, is a Program Manager for SQL Server on Linux.  He is working on the Linux experience, testing and capabilities surrounding SQL Server WAL and ACID capabilities.  Some of the areas he is currently looking into are as follows (as always these a goals we would like to achieve

As of this post, XFS on Linux 4.18 or newer is Fua optimized and we have also reached out to additional, Linux file system committers.  For example, the EXT4 committers are aware of the changes and evaluating the changes.

Odds and Ends

Updates and changes like this tend to expose some odds and ends.

Checkpoint Record Flush

SQL Server creates a log record (LOP_BUF_WRITE) to assist recovery. The record has an array of page ids written to disk. The checkpoint flush record has a limited size and is written whenever it becomes full.  The flushing of the checkpoint record requires FlushFileBuffers when forced flush behavior is enabled to maintain durability.

All Data Files For Same Database Flushed

Forced flush loops over all database files in the target database and issues FlushFileBuffers to complete a checkpoint. The data files may be on the same block device and the secondary flushes may be redundant but are inexpensive. Future SQL Server updates may optimize this behavior.

TEMPDB Is Faster

You may find that TEMPDB is faster using -T3982. TEMPDB is always rebuilt upon SQL Server restart and forced flush removes the FILE_FLAG_WRITE_THOUGH requirement of TEMPDB.  Since TEMPDB does not need to issue checkpoints, FlushFileBuffers may be skipped.  In fact, our performance team is evaluating a SQL Server 2019 change that never opens the TEMPDB file with the FILE_FLAG_WRITE_THROUGH.

RWF_ODSYNC

Another choice, provided in the Linux Fall 2017 Update, allows individual I/O requests (io_submit) to set the RWF_ODSYNC flag, forcing O_DSYNC for individual write requests. SQL Server does not use RWF_ODSYNC.

Simple Test and Results

IOPerfTest

SCSI Fua Write Command Reference [write(10), write(12), and write(16)]

https://www.seagate.com/staticfiles/support/disc/manuals/scsi/100293068a.pdf

Related References

• https://blogs.msdn.microsoft.com/psssql/2013/06/03/how-it-works-the-io-path-sql-server-running-in-windows-azure-virtual-machine-iaas
• https://blogs.msdn.microsoft.com/bobsql/2018/12/10/sql-server-instant-file-initialization-setfilevaliddata-windows-vs-fallocate-linux
• https://blogs.msdn.microsoft.com/bobsql/2018/07/31/sql-server-on-linux-how-is-delete-on-close-handled
• https://blogs.msdn.microsoft.com/bobsql/2017/01/31/sql-server-on-linux-scattergather-vectored-io
• https://blogs.msdn.microsoft.com/psssql/2010/03/24/how-it-works-bob-dorrs-sql-server-io-presentation
• https://blogs.msdn.microsoft.com/psssql/2011/01/07/discussion-about-sql-server-io
• https://technet.microsoft.com/en-us/library/cc966412.aspx
• https://blogs.msdn.microsoft.com/psssql/2008/11/12/how-it-works-sqliosim-running-average-target-duration-discarded-buffers
• https://blogs.msdn.microsoft.com/psssql/2008/04/04/sqliosim-is-not-an-io-performance-tuning-tool
• https://support.microsoft.com/en-us/help/231619/how-to-use-the-sqliosim-utility-to-simulate-sql-server-activity-on-a-d
• https://blogs.msdn.microsoft.com/psssql/2007/02/21/sql-server-urban-legends-discussed

Robert Dorr – SQL Server - Principle Software Engineer
Editing Assistance – Nathan Dorr

I was asked to validate  the SQL Server behavior in light of the error condition involving fsync/fdatasync outlined in this article. 

An example of the error: The application does a buffered write and receives success. (This means data can be stored in file system cache, non-stable media.)  An fsync/fdatasync is used to make sure the data is stored in stable media.  The stable media write(s) occurs during the sync operation and can fail for any number of reasons (out of disk space, lost connectivity to SAN, etc.) reporting an EIO error.

The issue, as described in the link, is that the sync returns the error but may clear the state of the cached pages.  The next sync returns ESUCCESS, meaning the the write(s), which were in cache, do not flush to stable media but applications were told they did.

Assume the database application opens the backup file, allowing file system caching (~_O_DIRECT.) SQL Server does NOT do this, in fact I made sure SQL Server on Linux does not allow this to occur.

· Backup is started using buffered I/O
· Writes occur to file system cache as the backup is progressing
· An external call to sync occurs causing file system cache writes to take place
· The sync encounters a write error on a buffer belonging to the backup.  The sync was called by an application outside database application so the database application is unaware of the failure.
· At the end of backup the database application issues fdatasync and instead of returning EIO, ESUCCESS is returned.
· A successful fdatasync indicates a hardened to stable media backup and the database application then truncates the inactive part of the transaction log.

The database no longer has the proper log records to perform transaction operations or recovery and the backup file, once thought to be valid, is invalid.

This issue does NOT impact SQL Server database, log and backup files.  SQL Server opens these file types using O_DIRECT to bypass file system cache. Even when SQL Server on Linux is running in Forced Flush mode the files are opened with O_DIRECT, avoiding the issue. 

Bob Dorr - Principal Software Engineer SQL Server